-----============= acceptance-small: sanityn ============----- Fri Sep 11 10:08:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg432-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 10:09:05 (1789135745) === Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Mount client oleg432-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg432-client.virtnet: executing check_config_client /mnt/lustre oleg432-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg432-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa091d940c800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa091e040e800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091d940c800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091e040e800.idle_timeout=debug disable quota as required oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanityn: finish setup 10:10:01 (1789135801) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200734 s, 52.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 10:10:07 (1789135807) PASS 0 (11s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 10:10:17 (1789135817) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 10:10:25 (1789135825) -rw-r--r-- 1 root root 0 Sep 11 10:10 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 10:10:32 (1789135832) -rw-r--r-- 1 root root 0 Sep 11 10:10 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 10:10:40 (1789135840) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (9s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 10:10:49 (1789135849) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (10s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 10:10:59 (1789135859) total 1 -rwxrwxrwx 1 root root 0 Sep 11 10:10 f2a -rwxrwxrwx 1 root root 0 Sep 11 10:10 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 11 10:10 f2a -rwxrwxrwx 1 root root 0 Sep 11 10:10 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0186047 s, 27.5 kB/s PASS 2e (10s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 10:11:09 (1789135869) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 10:11:14 (1789135874) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.4707 s, 4.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 10:11:23 (1789135883) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 10:11:32 (1789135892) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 10:11:40 (1789135900) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 10:11:47 (1789135907) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 10:11:56 (1789135916) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 10:12:04 (1789135924) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 10:12:12 (1789135932) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 10:12:20 (1789135940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669774 s, 15.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0981755 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 10:12:28 (1789135948) 1+0 records in 1+0 records out 1 byte copied, 0.0169363 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0237013 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.018473 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0158323 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00804401 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0178398 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0208319 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0218805 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0220518 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0134425 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00781267 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0106482 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 10:12:35 (1789135955) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0147771 s, 208 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248295 s, 165 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000191614 s, 16.0 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 10:12:42 (1789135962) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 10:12:50 (1789135970) warning: '-runtime' deprecated, use '-t runtime' instead - link 292 (time 1789135983.37 total 10.01 last 29.17) - link 565 (time 1789135993.38 total 20.02 last 27.28) - link 864 (time 1789136003.39 total 30.02 last 29.88) - link 1193 (time 1789136013.40 total 40.04 last 32.84) - link 1528 (time 1789136023.42 total 50.05 last 33.46) - link 1860 (time 1789136033.43 total 60.07 last 33.15) - link 2169 (time 1789136043.43 total 70.07 last 30.89) - link 2474 (time 1789136053.44 total 80.08 last 30.47) - link 2817 (time 1789136063.47 total 90.11 last 34.21) total: 3142 link in 100.01 seconds: 31.42 ops/second using seed 594553424 running for 100 seconds - stat 4286 (time 1789135985 ; total 11 ; last 11) - stat 10000 (time 1789135995 ; total 21 ; last 10) - stat 14941 (time 1789136006 ; total 32 ; last 11) - stat 20000 (time 1789136016 ; total 42 ; last 10) - stat 24859 (time 1789136027 ; total 53 ; last 11) - stat 27506 (time 1789136038 ; total 64 ; last 11) - stat 30000 (time 1789136047 ; total 73 ; last 9) - stat 32594 (time 1789136058 ; total 84 ; last 11) - stat 35310 (time 1789136069 ; total 95 ; last 11) total: 36871 stats in 100 seconds: 368.709991 stats/second - unlinked 0 (time 1789136024 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3142) error: No such file or directory total: 3141 unlinks in 87 seconds: 36.103447 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21647) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21649) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 10:15:16 (1789136116) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 10:15:27 (1789136127) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 22834 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 10:15:33 (1789136133) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 23414 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 10:15:40 (1789136140) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23992 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 10:15:46 (1789136146) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24575 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 10:15:54 (1789136154) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 25157 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 10:16:04 (1789136164) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (57s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 10:17:00 (1789136220) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.451459 s, 18.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.472776 s, 17.7 MB/s Chance of close/open is 1 in 50 Seed set to 6225 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x6e375c 000001[0] 1789136225.788248 trunc from 00000000 to 0x6e375b (0x6e375c bytes) 000002[0] 1789136225.807816 write 0x138fcf thru 0x14220d (0x923f bytes) truncating to largest ever: 0x8706e7 000005[0] 1789136225.881243 trunc from 0x6e375c to 0x8706e6 (0x18cf8b bytes) 000010[1] 1789136225.995087 write 0x1982fc thru 0x19bf1c (0x3c21 bytes) 000013[0] 1789136226.091759 read 0x760faf thru 0x7696a3 (0x86f5 bytes) 000014[1] 1789136226.116291 trunc from 0x8706e7 to 0x183f5f (0x6ec787 bytes) 000017[0] 1789136226.218345 write 0x2141b4 thru 0x217c5a (0x3aa7 bytes) truncating to largest ever: 0x8ad4e0 000018[0] 1789136226.274928 trunc from 0x217c5b to 0x8ad4df (0x695885 bytes) 000019[1] 1789136226.357848 write 0x5c4373 thru 0x5d1489 (0xd117 bytes) 000021[1] 1789136226.457065 mapread 0x56dc01 thru 0x5745fb (0x69fb bytes) 000023[0] 1789136226.542995 write 0x8afe46 thru 0x8bf3ce (0xf589 bytes) truncating to largest ever: 0x989bb3 000024[0] 1789136226.572845 trunc from 0x8bf3cf to 0x989bb2 (0xca7e4 bytes) 000025[0] 1789136226.628489 mapread 0x30d0e7 thru 0x31481e (0x7738 bytes) 000028[1] 1789136226.693092 read 0x3001cc thru 0x30be25 (0xbc5a bytes) 000030[1] 1789136226.765219 write 0x966d27 thru 0x96e4ca (0x77a4 bytes) 000033[1] 1789136226.877106 mapwrite 0x8d38b2 thru 0x8d63dd (0x2b2c bytes) 000034[0] 1789136227.054767 mapread 0x16891c thru 0x176940 (0xe025 bytes) 000036[1] 1789136227.139811 mapwrite 0x28a8d3 thru 0x28efa5 (0x46d3 bytes) 000039[0] 1789136227.320653 trunc from 0x989bb3 to 0x5a0de7 (0x3e8dcb bytes) 000042[1] 1789136227.395903 write 0x2f8beb thru 0x3007c0 (0x7bd6 bytes) 000043[1] 1789136227.435614 mapwrite 0x45a832 thru 0x462084 (0x7853 bytes) 000045[1] 1789136227.577168 read 0x036027 thru 0x03d386 (0x7360 bytes) 000046[0] 1789136227.587618 mapwrite 0x15cef1 thru 0x1673a4 (0xa4b4 bytes) 000048[1] 1789136227.747993 trunc from 0x5a0de8 to 0x6ac5da (0x10b7f3 bytes) 000049[1] 1789136227.817091 mapwrite 0x5e7a1c thru 0x5f6a68 (0xf04d bytes) 000050[0] 1789136227.925957 read 0x28ce36 thru 0x290d04 (0x3ecf bytes) 000052[0] 1789136227.972707 trunc from 0x6ac5db to 0x901407 (0x254e2d bytes) 000053[1] 1789136228.019103 write 0x071853 thru 0x0733e3 (0x1b91 bytes) 000054[1] 1789136228.056597 read 0x8fb328 thru 0x8fd4b1 (0x218a bytes) 000055[1] 1789136228.077442 read 0x86a28e thru 0x86f5d8 (0x534b bytes) 000056[0] 1789136228.100605 trunc from 0x901408 to 0x0f6114 (0x80b2f3 bytes) 000057[0] 1789136228.157167 mapread 0x0710fd thru 0x07c805 (0xb709 bytes) 000058[1] 1789136228.190075 trunc from 0x0f6115 to 0x85995a (0x763846 bytes) 000059[1] 1789136228.238161 mapread 0x07c336 thru 0x087aa7 (0xb772 bytes) 000060[1] 1789136228.268300 write 0x530f19 thru 0x5339b8 (0x2aa0 bytes) 000061[0] 1789136228.302342 read 0x329f6e thru 0x330c56 (0x6ce9 bytes) 000062[1] 1789136228.327803 mapread 0x29ae9a thru 0x29c2a3 (0x140a bytes) 000064[0] 1789136228.345090 mapwrite 0x40dc5e thru 0x419562 (0xb905 bytes) 000067[0] 1789136228.450097 mapwrite 0x34b370 thru 0x34f977 (0x4608 bytes) 000075[1] 1789136228.591202 read 0x3d9ce9 thru 0x3dbad7 (0x1def bytes) 000077[1] 1789136228.630684 mapwrite 0x97e6e3 thru 0x9829e7 (0x4305 bytes) 000078[0] 1789136228.727600 mapwrite 0x4b6203 thru 0x4bb6f4 (0x54f2 bytes) 000080[1] 1789136228.863929 trunc from 0x9829e8 to 0x8f8855 (0x8a192 bytes) 000083[0] 1789136229.017826 mapwrite 0x976ade thru 0x9789c0 (0x1ee3 bytes) 000084[0] 1789136229.197593 trunc from 0x9789c1 to 0x1d68d5 (0x7a20eb bytes) 000085[0] 1789136229.294855 write 0x872c16 thru 0x87a5e6 (0x79d1 bytes) 000087[0] 1789136229.390065 trunc from 0x87a5e7 to 0x0f55bb (0x78502b bytes) 000089[1] 1789136229.473232 write 0x50a62e thru 0x51668c (0xc05f bytes) 000096[0] 1789136229.734125 mapread 0x25e680 thru 0x260b12 (0x2493 bytes) 000101[0] 1789136230.030138 trunc from 0x51668d to 0x1b1f43 (0x364749 bytes) 000103[0] 1789136230.190058 write 0x712485 thru 0x720246 (0xddc2 bytes) 000105[1] 1789136230.260046 read 0x562108 thru 0x56ff5c (0xde55 bytes) 000107[1] 1789136230.345217 trunc from 0x720247 to 0x966a9e (0x246858 bytes) 000108[0] 1789136230.473764 mapwrite 0x44064f thru 0x44d8bc (0xd26e bytes) 000111[0] 1789136230.828641 mapwrite 0x182520 thru 0x1872c8 (0x4da9 bytes) 000113[1] 1789136231.173194 mapread 0x550f78 thru 0x551477 (0x500 bytes) 000116[1] 1789136231.330052 write 0x4d105e thru 0x4d93e5 (0x8388 bytes) 000119[0] 1789136231.456642 read 0x396e08 thru 0x39b58b (0x4784 bytes) 000121[0] 1789136231.503044 mapwrite 0x8cf9f8 thru 0x8d12c6 (0x18cf bytes) 000122[0] 1789136231.648855 read 0x5c4778 thru 0x5d3423 (0xecac bytes) 000124[1] 1789136231.684973 mapread 0x63b355 thru 0x63ef70 (0x3c1c bytes) 000139[0] 1789136231.921600 mapread 0x11f19f thru 0x1234a8 (0x430a bytes) 000142[0] 1789136231.949408 write 0x5ba5b4 thru 0x5c56f6 (0xb143 bytes) 000143[0] 1789136231.998429 mapread 0x5842e0 thru 0x58a842 (0x6563 bytes) 000144[0] 1789136232.036240 read 0x6c2349 thru 0x6cc5f4 (0xa2ac bytes) 000145[0] 1789136232.066951 trunc from 0x966a9f to 0x2bbfb7 (0x6aaae7 bytes) 000146[0] 1789136232.101855 trunc from 0x2bbfb8 to 0x0df151 (0x1dce66 bytes) 000148[1] 1789136232.123952 trunc from 0x0df152 to 0x9878d9 (0x8a8788 bytes) 000149[1] 1789136232.169837 write 0x7f0a7d thru 0x7f67bf (0x5d43 bytes) 000150[0] 1789136232.181777 read 0x9856a0 thru 0x9878d9 (0x223a bytes) 000151[0] 1789136232.379948 write 0x2993f8 thru 0x2a6e1c (0xda25 bytes) 000152[0] 1789136232.422943 mapwrite 0x4d68e1 thru 0x4d9d01 (0x3421 bytes) 000153[0] 1789136232.610098 mapwrite 0x18310f thru 0x188de0 (0x5cd2 bytes) 000156[0] 1789136232.989962 trunc from 0x9878da to 0x23af73 (0x74c966 bytes) 000159[0] 1789136233.053903 read 0x1194ab thru 0x127e44 (0xe99a bytes) 000160[0] 1789136233.132108 read 0x0cd7b6 thru 0x0d8db0 (0xb5fb bytes) 000162[0] 1789136233.170043 mapread 0x05bae3 thru 0x0603f5 (0x4913 bytes) 000163[0] 1789136233.170979 mapwrite 0x4ba718 thru 0x4bc66f (0x1f58 bytes) 000165[1] 1789136233.338351 mapread 0x0066d8 thru 0x011ecd (0xb7f6 bytes) 000167[0] 1789136233.409064 read 0x1625e4 thru 0x1698b9 (0x72d6 bytes) 000168[1] 1789136233.430758 trunc from 0x4bc670 to 0x77dc1b (0x2c15ac bytes) 000173[0] 1789136233.579014 mapread 0x6bd956 thru 0x6cccf1 (0xf39c bytes) 000174[1] 1789136233.660470 trunc from 0x77dc1c to 0x8ac83d (0x12ec22 bytes) 000175[1] 1789136233.722684 mapread 0x4a3be7 thru 0x4a5447 (0x1861 bytes) 000180[1] 1789136233.825932 read 0x6ece5d thru 0x6fbe63 (0xf007 bytes) 000181[0] 1789136233.835281 read 0x347896 thru 0x34d17f (0x58ea bytes) 000186[1] 1789136233.941410 mapwrite 0x18d5db thru 0x19bd7e (0xe7a4 bytes) 000187[1] 1789136234.180536 write 0x9cf361 thru 0x9dc520 (0xd1c0 bytes) 000188[1] 1789136234.182343 mapwrite 0x2d6548 thru 0x2da721 (0x41da bytes) 000189[0] 1789136234.299687 mapwrite 0x46d241 thru 0x47a55c (0xd31c bytes) 000190[1] 1789136234.486317 mapwrite 0x4eb8ed thru 0x4f2812 (0x6f26 bytes) 000191[1] 1789136234.573988 write 0x2ca977 thru 0x2d986e (0xeef8 bytes) 000193[0] 1789136234.600924 mapread 0x68e620 thru 0x691e4b (0x382c bytes) 000194[1] 1789136234.645542 read 0x2f3a95 thru 0x2f8bd2 (0x513e bytes) 000195[1] 1789136234.668521 trunc from 0x9dc521 to 0x60d050 (0x3cf4d0 bytes) 000196[1] 1789136234.709065 read 0x351085 thru 0x352319 (0x1295 bytes) 000197[0] 1789136234.737890 mapwrite 0x11c45b thru 0x128187 (0xbd2d bytes) 000199[1] 1789136234.890974 write 0x81b341 thru 0x820816 (0x54d6 bytes) 000203[0] 1789136234.985645 trunc from 0x820817 to 0x461d05 (0x3beb11 bytes) 000204[0] 1789136235.054196 read 0x04628d thru 0x049dee (0x3b62 bytes) 000206[0] 1789136235.086664 write 0x35e321 thru 0x365a1d (0x76fd bytes) 000207[1] 1789136235.096960 mapwrite 0x8e8f38 thru 0x8f1f82 (0x904b bytes) 000208[1] 1789136235.226633 mapwrite 0x12e127 thru 0x1309e1 (0x28bb bytes) 000210[0] 1789136235.304921 write 0x923341 thru 0x930af8 (0xd7b8 bytes) 000211[1] 1789136235.351944 read 0x4a889f thru 0x4b7162 (0xe8c4 bytes) 000214[0] 1789136235.406409 trunc from 0x930af9 to 0x1c1607 (0x76f4f1 bytes) 000216[1] 1789136235.438070 mapwrite 0x70ebc3 thru 0x71246a (0x38a8 bytes) 000217[1] 1789136235.555186 write 0x8e9317 thru 0x8f328d (0x9f77 bytes) 000218[0] 1789136235.583850 read 0x506816 thru 0x50c475 (0x5c60 bytes) 000219[0] 1789136235.607458 trunc from 0x8f328e to 0x785d01 (0x16d58c bytes) 000220[1] 1789136235.672691 read 0x71a8be thru 0x7206aa (0x5ded bytes) 000222[1] 1789136235.700564 read 0x145cb8 thru 0x14ed4a (0x9093 bytes) 000225[1] 1789136235.741494 trunc from 0x785d02 to 0x71c0c5 (0x69c3c bytes) 000226[1] 1789136235.797923 mapwrite 0x7729f5 thru 0x77b9ab (0x8fb7 bytes) 000227[1] 1789136235.881396 mapwrite 0x5c6f58 thru 0x5cf2e0 (0x8389 bytes) 000229[1] 1789136235.937080 trunc from 0x77b9ac to 0x47397d (0x30802e bytes) 000230[0] 1789136235.953311 read 0x2c8d7f thru 0x2d24b7 (0x9739 bytes) 000231[1] 1789136235.977427 write 0x82dd33 thru 0x83da61 (0xfd2f bytes) 000232[1] 1789136236.017177 mapwrite 0x762786 thru 0x76807f (0x58fa bytes) 000236[0] 1789136236.123323 mapwrite 0x5b226f thru 0x5b33be (0x1150 bytes) 000237[1] 1789136236.206906 read 0x60c1e7 thru 0x61378b (0x75a5 bytes) 000242[1] 1789136236.292343 mapread 0x011288 thru 0x01f75a (0xe4d3 bytes) 000244[0] 1789136236.320165 trunc from 0x83da62 to 0x9480f7 (0x10a696 bytes) 000246[1] 1789136236.412415 trunc from 0x9480f8 to 0x61c0dd (0x32c01a bytes) 000247[1] 1789136236.487677 trunc from 0x61c0de to 0x7ab98b (0x18f8ae bytes) 000248[0] 1789136236.523688 mapread 0x5be0e9 thru 0x5c3ef3 (0x5e0b bytes) 000250[0] 1789136236.563366 trunc from 0x7ab98c to 0x2d26e7 (0x4d92a4 bytes) 000251[1] 1789136236.673531 mapread 0x284dee thru 0x2942db (0xf4ee bytes) 000253[1] 1789136236.778020 mapwrite 0x254e39 thru 0x259332 (0x44fa bytes) 000254[1] 1789136236.883621 read 0x271d1d thru 0x27ef7f (0xd263 bytes) 000255[1] 1789136236.964595 trunc from 0x2d26e8 to 0x58a3b7 (0x2b7cd0 bytes) 000256[1] 1789136237.003620 read 0x21946e thru 0x22431b (0xaeae bytes) 000258[1] 1789136237.004680 write 0x3a829f thru 0x3b79d4 (0xf736 bytes) 000261[1] 1789136237.085579 mapread 0x131073 thru 0x132a5e (0x19ec bytes) 000265[1] 1789136237.155367 read 0x4888ac thru 0x49744f (0xeba4 bytes) 000266[0] 1789136237.165816 read 0x49e746 thru 0x4ae3ad (0xfc68 bytes) 000268[1] 1789136237.189449 read 0x1aebb2 thru 0x1b8e50 (0xa29f bytes) 000269[1] 1789136237.190253 read 0x4ca861 thru 0x4cb62f (0xdcf bytes) 000270[1] 1789136237.208559 trunc from 0x58a3b8 to 0x1b9d8c (0x3d062b bytes) 000272[0] 1789136237.245979 write 0x287482 thru 0x290b43 (0x96c2 bytes) 000274[1] 1789136237.291454 read 0x03e556 thru 0x03f471 (0xf1c bytes) truncating to largest ever: 0x9ebd73 000275[1] 1789136237.375929 trunc from 0x290b44 to 0x9ebd72 (0x75b22f bytes) 000276[1] 1789136237.417689 trunc from 0x9ebd73 to 0x7510b6 (0x29acbc bytes) 000277[0] 1789136237.435835 mapwrite 0x0e7e44 thru 0x0f78ae (0xfa6b bytes) 000278[0] 1789136237.604835 read 0x26e4d5 thru 0x276574 (0x80a0 bytes) 000279[1] 1789136237.643040 write 0x4a2dff thru 0x4b0672 (0xd874 bytes) 000281[0] 1789136237.692780 trunc from 0x7510b7 to 0x1e629a (0x56ae1c bytes) 000284[1] 1789136237.758709 mapwrite 0x481426 thru 0x484daf (0x398a bytes) 000289[0] 1789136238.019034 trunc from 0x484db0 to 0x923fbb (0x49f20c bytes) 000291[0] 1789136238.076886 write 0x4ddb53 thru 0x4e5562 (0x7a10 bytes) 000293[0] 1789136238.155556 read 0x4b7983 thru 0x4b8ccd (0x134b bytes) 000294[1] 1789136238.210058 write 0x8264d7 thru 0x82bda8 (0x58d2 bytes) 000295[1] 1789136238.264589 write 0x4ca024 thru 0x4d45f8 (0xa5d5 bytes) 000297[1] 1789136238.421386 mapread 0x1e18c7 thru 0x1f181e (0xff58 bytes) 000299[1] 1789136238.595026 mapwrite 0x5fa84b thru 0x601c8c (0x7442 bytes) 000302[1] 1789136238.717588 mapwrite 0x8a3a18 thru 0x8a71fc (0x37e5 bytes) 000305[1] 1789136238.918929 mapwrite 0x0c7ee2 thru 0x0d7227 (0xf346 bytes) 000306[0] 1789136239.084383 mapwrite 0x81dad3 thru 0x822598 (0x4ac6 bytes) 000307[1] 1789136239.200942 mapwrite 0x96271b thru 0x962875 (0x15b bytes) 000308[1] 1789136239.346563 mapwrite 0x7e234e thru 0x7e5fcb (0x3c7e bytes) 000315[1] 1789136239.472422 trunc from 0x962876 to 0x3e8342 (0x57a533 bytes) 000318[1] 1789136239.541208 trunc from 0x3e8343 to 0x41ecfa (0x369b8 bytes) 000320[1] 1789136239.603610 read 0x1ecd70 thru 0x1f8ff6 (0xc287 bytes) 000324[0] 1789136239.694896 write 0x00175e thru 0x00211c (0x9bf bytes) 000325[0] 1789136239.733551 write 0x5ef718 thru 0x5f207c (0x2965 bytes) 000326[1] 1789136239.750534 read 0x5bb801 thru 0x5c8596 (0xcd96 bytes) 000327[0] 1789136239.789135 write 0x32b079 thru 0x3329cd (0x7955 bytes) 000328[1] 1789136239.822176 mapwrite 0x646099 thru 0x65370d (0xd675 bytes) 000330[0] 1789136239.996576 mapread 0x4dc3ce thru 0x4dc82b (0x45e bytes) 000331[0] 1789136240.052179 mapread 0x597360 thru 0x5a0e0b (0x9aac bytes) 000333[1] 1789136240.102655 write 0x3244fd thru 0x324df7 (0x8fb bytes) 000334[0] 1789136240.123701 mapwrite 0x097ffd thru 0x0a6cb7 (0xecbb bytes) 000340[0] 1789136240.313342 read 0x3b9294 thru 0x3c0108 (0x6e75 bytes) 000341[0] 1789136240.345539 read 0x0d40e4 thru 0x0dc1a1 (0x80be bytes) 000346[0] 1789136240.370469 trunc from 0x65370e to 0x12ce19 (0x5268f4 bytes) 000352[1] 1789136240.567353 mapread 0x121b06 thru 0x1246bb (0x2bb6 bytes) 000353[0] 1789136240.669600 mapwrite 0x1c4376 thru 0x1d3319 (0xefa4 bytes) 000355[0] 1789136240.924958 write 0x2f3822 thru 0x2f5136 (0x1915 bytes) 000357[1] 1789136240.963401 mapwrite 0x5801e1 thru 0x584c2d (0x4a4d bytes) 000359[1] 1789136241.100189 mapread 0x4bd9f9 thru 0x4cd632 (0xfc3a bytes) 000360[1] 1789136241.121754 trunc from 0x584c2e to 0x80d6e6 (0x288ab9 bytes) 000361[0] 1789136241.170993 read 0x67b688 thru 0x67de66 (0x27df bytes) 000364[1] 1789136241.219443 trunc from 0x80d6e7 to 0x7ee1f7 (0x1f4ef bytes) 000369[1] 1789136241.338645 trunc from 0x7ee1f8 to 0x1d2aaa (0x61b74d bytes) 000371[1] 1789136241.363834 read 0x036239 thru 0x038f22 (0x2cea bytes) 000372[0] 1789136241.398112 mapwrite 0x3c4139 thru 0x3c960d (0x54d5 bytes) 000375[1] 1789136241.537865 mapread 0x239927 thru 0x23d88f (0x3f69 bytes) 000376[0] 1789136241.621986 read 0x037c78 thru 0x04140c (0x9795 bytes) 000378[0] 1789136241.665881 trunc from 0x3c960e to 0x84a8a6 (0x481299 bytes) 000387[0] 1789136241.826799 read 0x811577 thru 0x81a41a (0x8ea4 bytes) 000388[0] 1789136241.860597 mapread 0x3263ef thru 0x32974a (0x335c bytes) 000391[1] 1789136241.944850 mapread 0x6dd6a3 thru 0x6e9fee (0xc94c bytes) 000393[0] 1789136242.086672 write 0x8cb645 thru 0x8ce092 (0x2a4e bytes) 000400[0] 1789136242.161851 mapwrite 0x8c08bb thru 0x8c7d58 (0x749e bytes) 000401[0] 1789136242.258570 mapwrite 0x30b05b thru 0x310ecb (0x5e71 bytes) 000402[1] 1789136242.328621 read 0x1feadf thru 0x2018f3 (0x2e15 bytes) 000407[1] 1789136242.356194 trunc from 0x8ce093 to 0x8d4159 (0x60c7 bytes) 000408[0] 1789136242.414461 write 0x85a0ff thru 0x86531f (0xb221 bytes) 000410[1] 1789136242.478814 write 0x4e1e96 thru 0x4f0a6b (0xebd6 bytes) 000411[1] 1789136242.504529 trunc from 0x8d415a to 0x1f1f5c (0x6e21fd bytes) 000416[1] 1789136242.617871 write 0x5fb415 thru 0x5fcdfe (0x19ea bytes) 000417[1] 1789136242.645378 write 0x1568aa thru 0x164d0f (0xe466 bytes) 000419[1] 1789136242.660396 trunc from 0x5fcdff to 0x3e1753 (0x21b6ab bytes) 000420[1] 1789136242.709259 write 0x476893 thru 0x483330 (0xca9e bytes) 000425[0] 1789136242.771852 trunc from 0x483331 to 0x7f6ab6 (0x373786 bytes) 000426[0] 1789136242.883404 write 0x382e3b thru 0x3910d9 (0xe29f bytes) 000427[1] 1789136242.963409 trunc from 0x7f6ab7 to 0x57cd7f (0x279d37 bytes) 000428[0] 1789136243.083248 read 0x1bf4b7 thru 0x1c2693 (0x31dd bytes) 000429[1] 1789136243.134252 read 0x52c06b thru 0x539e11 (0xdda7 bytes) 000431[1] 1789136243.144603 read 0x3a4198 thru 0x3afede (0xbd47 bytes) 000432[1] 1789136243.172041 read 0x4c7653 thru 0x4cad4b (0x36f9 bytes) 000433[1] 1789136243.197658 write 0x1305f0 thru 0x134998 (0x43a9 bytes) 000434[0] 1789136243.224154 mapwrite 0x834ec8 thru 0x8422e9 (0xd422 bytes) 000437[0] 1789136243.472683 read 0x4309f9 thru 0x43d76d (0xcd75 bytes) 000439[0] 1789136243.480881 mapwrite 0x226bbe thru 0x2294dc (0x291f bytes) 000440[1] 1789136243.543107 write 0x76f9e1 thru 0x77cee6 (0xd506 bytes) 000441[1] 1789136243.587680 read 0x2c2200 thru 0x2c9359 (0x715a bytes) 000442[0] 1789136243.611824 mapwrite 0x00778a thru 0x0143d9 (0xcc50 bytes) 000443[0] 1789136243.833233 mapwrite 0x8e1503 thru 0x8e8e80 (0x797e bytes) 000444[0] 1789136243.939878 mapwrite 0x1cd760 thru 0x1ce1cf (0xa70 bytes) truncating to largest ever: 0x9ee4c4 000446[1] 1789136244.055509 trunc from 0x8e8e81 to 0x9ee4c3 (0x105643 bytes) 000448[1] 1789136244.139251 read 0x68b9f4 thru 0x695e62 (0xa46f bytes) 000450[1] 1789136244.154248 write 0x4c9b9d thru 0x4d091b (0x6d7f bytes) 000451[1] 1789136244.181827 trunc from 0x9ee4c4 to 0x54a4ae (0x4a4015 bytes) 000453[0] 1789136244.259841 trunc from 0x54a4af to 0x8d8b1e (0x38e670 bytes) 000454[0] 1789136244.395462 mapwrite 0x1d15c2 thru 0x1da6f5 (0x9134 bytes) 000461[1] 1789136244.564592 mapread 0x12c653 thru 0x133c2d (0x75db bytes) 000462[1] 1789136244.624365 read 0x293e95 thru 0x2a387f (0xf9eb bytes) 000464[0] 1789136244.647222 mapread 0x3a409a thru 0x3b20cf (0xe036 bytes) 000467[1] 1789136244.687967 read 0x358975 thru 0x35d19e (0x482a bytes) 000469[0] 1789136244.719598 write 0x66229f thru 0x662d32 (0xa94 bytes) 000471[1] 1789136244.735336 trunc from 0x8d8b1f to 0x8326e7 (0xa6437 bytes) 000472[1] 1789136244.885166 read 0x051bcd thru 0x05fafb (0xdf2f bytes) 000474[0] 1789136244.902462 read 0x7aafc8 thru 0x7ae7e1 (0x381a bytes) 000476[0] 1789136244.975257 mapwrite 0x84c95f thru 0x856b21 (0xa1c3 bytes) 000477[1] 1789136245.110391 mapread 0x187f89 thru 0x19394b (0xb9c3 bytes) 000478[1] 1789136245.188121 mapwrite 0x852c25 thru 0x858592 (0x596e bytes) 000480[0] 1789136245.332650 trunc from 0x858593 to 0x57f50e (0x2d9084 bytes) 000482[1] 1789136245.455312 write 0x0b83e6 thru 0x0bc2da (0x3ef5 bytes) 000483[1] 1789136245.530862 read 0x40bbfa thru 0x419523 (0xd92a bytes) 000484[0] 1789136245.559354 trunc from 0x57f50f to 0x15a18c (0x425382 bytes) 000485[1] 1789136245.634495 read 0x076c44 thru 0x077c56 (0x1013 bytes) 000487[0] 1789136245.686390 mapwrite 0x9167e7 thru 0x9216d7 (0xaef1 bytes) 000489[1] 1789136246.038171 write 0x5214d4 thru 0x52cd78 (0xb8a5 bytes) 000491[1] 1789136246.163488 mapread 0x7ddc77 thru 0x7e8f28 (0xb2b2 bytes) 000492[1] 1789136246.274541 mapread 0x27f9e0 thru 0x28e269 (0xe88a bytes) 000494[0] 1789136246.380378 write 0x51a5f6 thru 0x51ad93 (0x79e bytes) 000499[1] 1789136246.492262 trunc from 0x9216d8 to 0x85c194 (0xc5543 bytes) 000500[0] 1789136246.541598 trunc from 0x85c195 to 0x4d21ef (0x389fa5 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6246 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000011[0] 1789136247.497635 write_OD 0x592000 thru 0x59cfff (0xb000 bytes) 000012[0] 1789136247.615994 read_OD 0x0d8000 thru 0x0defff (0x7000 bytes) 000015[0] 1789136247.632881 read_OD 0x40b000 thru 0x410fff (0x6000 bytes) truncating to largest ever: 0x6d571e 000016[1] 1789136247.644404 trunc from 0x59d000 to 0x6d571d (0x13871e bytes) 000018[0] 1789136247.687026 read_OD 0x105000 thru 0x112fff (0xe000 bytes) 000019[0] 1789136247.716510 trunc from 0x6d571e to 0x6ba844 (0x1aed9 bytes) truncating to largest ever: 0x9ad31c 000020[1] 1789136247.747620 trunc from 0x6ba845 to 0x9ad31b (0x2f2ad7 bytes) 000022[0] 1789136247.800053 mapread 0x831000 thru 0x83ef76 (0xdf77 bytes) 000023[1] 1789136247.847646 trunc from 0x9ad31c to 0x1dba27 (0x7d18f4 bytes) 000026[1] 1789136247.939042 mapwrite 0x341000 thru 0x34939c (0x839d bytes) 000028[0] 1789136248.068765 trunc from 0x34939d to 0x0abb87 (0x29d815 bytes) 000029[1] 1789136248.111200 write_OD 0x2af000 thru 0x2b9fff (0xb000 bytes) 000030[1] 1789136248.186050 write_OD 0x89e000 thru 0x8acfff (0xf000 bytes) 000031[1] 1789136248.249800 trunc from 0x8ad000 to 0x109150 (0x7a3eaf bytes) 000032[0] 1789136248.294161 write_OD 0x503000 thru 0x50cfff (0xa000 bytes) 000033[1] 1789136248.355537 trunc from 0x50d000 to 0x76a327 (0x25d328 bytes) 000035[1] 1789136248.422273 read_OD 0x64b000 thru 0x652fff (0x8000 bytes) 000037[1] 1789136248.461083 read_OD 0x516000 thru 0x518fff (0x3000 bytes) 000043[0] 1789136248.473134 write_OD 0x9eb000 thru 0x9f0fff (0x6000 bytes) 000046[1] 1789136248.570497 mapwrite 0x13a000 thru 0x1479e2 (0xd9e3 bytes) 000047[1] 1789136248.725609 mapread 0x9aa000 thru 0x9b60f0 (0xc0f1 bytes) 000048[1] 1789136248.760880 mapread 0x411000 thru 0x41bca3 (0xaca4 bytes) 000050[0] 1789136248.810615 write_OD 0x4b5000 thru 0x4b7fff (0x3000 bytes) 000053[1] 1789136248.898210 read_OD 0x612000 thru 0x612fff (0x1000 bytes) 000054[1] 1789136248.910774 mapwrite 0x207000 thru 0x214a68 (0xda69 bytes) 000056[1] 1789136249.091395 mapread 0x193000 thru 0x1a262e (0xf62f bytes) 000058[1] 1789136249.199576 read_OD 0x1c1000 thru 0x1ccfff (0xc000 bytes) 000059[1] 1789136249.236359 read_OD 0x5d2000 thru 0x5d9fff (0x8000 bytes) 000061[1] 1789136249.274323 mapread 0x524000 thru 0x52b382 (0x7383 bytes) 000067[0] 1789136249.329763 mapread 0x426000 thru 0x42fdba (0x9dbb bytes) 000068[0] 1789136249.350498 write_OD 0x3cb000 thru 0x3d1fff (0x7000 bytes) 000072[1] 1789136249.475564 mapwrite 0x648000 thru 0x64e96a (0x696b bytes) 000074[1] 1789136249.726938 mapwrite 0x576000 thru 0x57c5a0 (0x65a1 bytes) 000076[0] 1789136249.914903 write_OD 0x2f4000 thru 0x2f5fff (0x2000 bytes) 000082[1] 1789136250.013138 write_OD 0x23a000 thru 0x243fff (0xa000 bytes) 000084[1] 1789136250.128827 mapwrite 0x1a6000 thru 0x1a6908 (0x909 bytes) 000085[1] 1789136250.252578 mapwrite 0x1ed000 thru 0x1eedfb (0x1dfc bytes) 000087[0] 1789136250.442130 read_OD 0x5e4000 thru 0x5eafff (0x7000 bytes) 000089[1] 1789136250.484311 read_OD 0x960000 thru 0x966fff (0x7000 bytes) 000093[1] 1789136250.500102 trunc from 0x9f1000 to 0x36b0d4 (0x685f2b bytes) 000094[0] 1789136250.549808 mapread 0x163000 thru 0x16a38b (0x738c bytes) 000095[0] 1789136250.609088 trunc from 0x36b0d5 to 0x701936 (0x396862 bytes) 000096[1] 1789136250.662508 trunc from 0x701937 to 0x4e6333 (0x21b603 bytes) 000098[1] 1789136250.711073 mapwrite 0x47f000 thru 0x484391 (0x5392 bytes) 000100[1] 1789136250.785800 read_OD 0x12f000 thru 0x135fff (0x7000 bytes) 000102[1] 1789136250.809254 write_OD 0x7c7000 thru 0x7d4fff (0xe000 bytes) 000103[0] 1789136250.878495 mapread 0x2fd000 thru 0x30a590 (0xd591 bytes) 000104[0] 1789136250.922763 read_OD 0x77c000 thru 0x77efff (0x3000 bytes) 000105[0] 1789136250.932107 trunc from 0x7d5000 to 0x4435f4 (0x391a0b bytes) 000106[1] 1789136251.009828 mapread 0x231000 thru 0x235f37 (0x4f38 bytes) 000108[1] 1789136251.081102 write_OD 0x8d4000 thru 0x8d8fff (0x5000 bytes) 000110[0] 1789136251.233751 read_OD 0x897000 thru 0x8a0fff (0xa000 bytes) 000114[0] 1789136251.244990 read_OD 0x7fa000 thru 0x801fff (0x8000 bytes) 000117[1] 1789136251.280223 mapwrite 0x2d0000 thru 0x2d0771 (0x772 bytes) 000119[1] 1789136251.425071 mapwrite 0x925000 thru 0x926381 (0x1382 bytes) 000120[1] 1789136251.524194 trunc from 0x926382 to 0x879c85 (0xac6fc bytes) 000123[0] 1789136251.573334 mapwrite 0x4a3000 thru 0x4aa95e (0x795f bytes) 000129[1] 1789136251.719311 trunc from 0x879c86 to 0x1350ba (0x744bcb bytes) 000132[1] 1789136251.766503 write_OD 0x71f000 thru 0x71ffff (0x1000 bytes) 000135[1] 1789136251.829876 trunc from 0x720000 to 0x7c7fdc (0xa7fdd bytes) 000136[0] 1789136251.897598 mapwrite 0x265000 thru 0x266b80 (0x1b81 bytes) 000138[0] 1789136252.207062 write_OD 0x887000 thru 0x894fff (0xe000 bytes) 000140[1] 1789136252.309041 mapread 0x1ea000 thru 0x1f4ebb (0xaebc bytes) 000143[1] 1789136252.331034 mapread 0x167000 thru 0x1689d5 (0x19d6 bytes) 000144[1] 1789136252.346346 write_OD 0x296000 thru 0x2a2fff (0xd000 bytes) 000145[1] 1789136252.477263 write_OD 0x598000 thru 0x5a0fff (0x9000 bytes) 000147[1] 1789136252.585375 write_OD 0x6dc000 thru 0x6e2fff (0x7000 bytes) 000148[0] 1789136252.628474 read_OD 0x0eb000 thru 0x0f7fff (0xd000 bytes) 000149[0] 1789136252.641820 mapwrite 0x6dd000 thru 0x6e4135 (0x7136 bytes) 000151[0] 1789136252.764959 mapread 0x092000 thru 0x09a453 (0x8454 bytes) 000153[1] 1789136252.797049 mapwrite 0x7e8000 thru 0x7f1326 (0x9327 bytes) 000154[0] 1789136253.032236 mapwrite 0x44a000 thru 0x459339 (0xf33a bytes) 000157[0] 1789136253.271261 read_OD 0x3da000 thru 0x3e7fff (0xe000 bytes) 000159[0] 1789136253.295074 mapwrite 0x122000 thru 0x1314b7 (0xf4b8 bytes) 000161[0] 1789136253.386508 trunc from 0x895000 to 0x78b5ca (0x109a35 bytes) 000163[0] 1789136253.402632 read_OD 0x079000 thru 0x079fff (0x1000 bytes) 000164[0] 1789136253.421476 mapwrite 0x4a9000 thru 0x4b529d (0xc29e bytes) 000165[1] 1789136253.505908 mapread 0x700000 thru 0x70fec4 (0xfec5 bytes) 000166[0] 1789136253.535943 write_OD 0x0cf000 thru 0x0d5fff (0x7000 bytes) 000168[1] 1789136253.588705 trunc from 0x78b5cb to 0x0b4290 (0x6d733a bytes) 000171[0] 1789136253.639963 write_OD 0x0a7000 thru 0x0a8fff (0x2000 bytes) 000173[1] 1789136253.696215 read_OD 0x08a000 thru 0x090fff (0x7000 bytes) 000175[1] 1789136253.703450 read_OD 0x006000 thru 0x013fff (0xe000 bytes) 000176[1] 1789136253.710766 read_OD 0x089000 thru 0x093fff (0xb000 bytes) 000181[0] 1789136253.721081 trunc from 0x0b4291 to 0x7f4a51 (0x7407c1 bytes) 000182[1] 1789136253.756125 write_OD 0x738000 thru 0x741fff (0xa000 bytes) 000184[1] 1789136253.810128 trunc from 0x7f4a52 to 0x5df519 (0x215538 bytes) 000185[1] 1789136253.849740 mapread 0x4a5000 thru 0x4b0a22 (0xba23 bytes) 000187[0] 1789136253.937939 mapread 0x2c3000 thru 0x2c9a18 (0x6a19 bytes) 000189[1] 1789136254.026237 read_OD 0x364000 thru 0x370fff (0xd000 bytes) 000190[1] 1789136254.037135 trunc from 0x5df51a to 0x529811 (0xb5d08 bytes) 000191[1] 1789136254.107776 read_OD 0x350000 thru 0x35cfff (0xd000 bytes) 000193[1] 1789136254.143581 mapread 0x01e000 thru 0x023346 (0x5347 bytes) 000196[1] 1789136254.188356 trunc from 0x529812 to 0x419bb9 (0x10fc58 bytes) truncating to largest ever: 0x9b39e9 000197[0] 1789136254.246104 trunc from 0x419bba to 0x9b39e8 (0x599e2f bytes) 000199[0] 1789136254.323537 trunc from 0x9b39e9 to 0x0d73c3 (0x8dc625 bytes) 000200[0] 1789136254.363786 mapwrite 0x45d000 thru 0x45fd38 (0x2d39 bytes) 000201[1] 1789136254.448001 read_OD 0x014000 thru 0x021fff (0xe000 bytes) 000202[0] 1789136254.486035 write_OD 0x114000 thru 0x11dfff (0xa000 bytes) 000203[1] 1789136254.555568 write_OD 0x4ab000 thru 0x4b2fff (0x8000 bytes) 000209[1] 1789136254.665808 trunc from 0x4b3000 to 0x21d5c2 (0x295a3d bytes) 000213[0] 1789136254.782684 trunc from 0x21d5c3 to 0x5e8687 (0x3cb0c5 bytes) 000214[0] 1789136254.817116 mapwrite 0x77d000 thru 0x788a11 (0xba12 bytes) 000215[1] 1789136254.936506 mapread 0x73a000 thru 0x748f50 (0xef51 bytes) 000217[0] 1789136255.012387 read_OD 0x41a000 thru 0x422fff (0x9000 bytes) 000218[1] 1789136255.032383 mapread 0x479000 thru 0x47c5ec (0x35ed bytes) 000219[1] 1789136255.056441 write_OD 0x7d3000 thru 0x7d3fff (0x1000 bytes) 000221[1] 1789136255.117577 read_OD 0x46d000 thru 0x479fff (0xd000 bytes) 000222[0] 1789136255.129005 write_OD 0x3ab000 thru 0x3adfff (0x3000 bytes) 000224[0] 1789136255.183681 read_OD 0x3c9000 thru 0x3d5fff (0xd000 bytes) 000226[0] 1789136255.193003 write_OD 0x47c000 thru 0x487fff (0xc000 bytes) 000227[0] 1789136255.254696 mapwrite 0x13e000 thru 0x14cfb5 (0xefb6 bytes) 000228[1] 1789136255.361434 trunc from 0x7d4000 to 0x39ae05 (0x4391fa bytes) 000229[1] 1789136255.399510 mapread 0x07c000 thru 0x081765 (0x5766 bytes) 000234[0] 1789136255.452616 mapwrite 0x20a000 thru 0x2108f7 (0x68f8 bytes) 000235[1] 1789136255.556088 write_OD 0x7ff000 thru 0x809fff (0xb000 bytes) 000236[1] 1789136255.608310 mapread 0x641000 thru 0x64a183 (0x9184 bytes) 000238[1] 1789136255.629155 read_OD 0x4bf000 thru 0x4c1fff (0x3000 bytes) 000240[0] 1789136255.634652 mapwrite 0x460000 thru 0x461b64 (0x1b65 bytes) 000242[0] 1789136255.728846 mapwrite 0x9f5000 thru 0x9ff8d1 (0xa8d2 bytes) 000250[0] 1789136255.983797 mapread 0x38a000 thru 0x39781e (0xd81f bytes) 000251[0] 1789136256.045255 mapread 0x66f000 thru 0x671e82 (0x2e83 bytes) 000252[0] 1789136256.085459 mapwrite 0x9b7000 thru 0x9c29c5 (0xb9c6 bytes) 000256[0] 1789136256.169195 write_OD 0x138000 thru 0x13afff (0x3000 bytes) 000260[1] 1789136256.218585 mapwrite 0x0bb000 thru 0x0c5136 (0xa137 bytes) 000262[0] 1789136256.299196 mapread 0x7e1000 thru 0x7e6c4c (0x5c4d bytes) 000263[0] 1789136256.338566 read_OD 0x7a0000 thru 0x7a3fff (0x4000 bytes) 000266[1] 1789136256.349148 mapwrite 0x0ce000 thru 0x0db509 (0xd50a bytes) 000268[1] 1789136256.470602 mapread 0x412000 thru 0x41c566 (0xa567 bytes) 000269[0] 1789136256.545616 trunc from 0x9ff8d2 to 0x8e1f43 (0x11d98e bytes) 000271[0] 1789136256.622647 read_OD 0x015000 thru 0x019fff (0x5000 bytes) 000272[1] 1789136256.693272 trunc from 0x8e1f44 to 0x740928 (0x1a161b bytes) 000275[1] 1789136256.814262 mapread 0x33f000 thru 0x341f25 (0x2f26 bytes) 000278[0] 1789136256.849680 write_OD 0x065000 thru 0x06cfff (0x8000 bytes) 000279[0] 1789136256.950516 write_OD 0x6a6000 thru 0x6aefff (0x9000 bytes) 000283[0] 1789136257.038812 mapread 0x1f5000 thru 0x1f9711 (0x4712 bytes) 000284[1] 1789136257.079528 mapread 0x037000 thru 0x04693a (0xf93b bytes) 000285[0] 1789136257.134524 mapread 0x4a2000 thru 0x4a2e9a (0xe9b bytes) 000286[0] 1789136257.142784 read_OD 0x3c9000 thru 0x3d7fff (0xf000 bytes) 000291[1] 1789136257.168530 read_OD 0x6df000 thru 0x6e2fff (0x4000 bytes) 000294[0] 1789136257.183977 read_OD 0x720000 thru 0x721fff (0x2000 bytes) 000300[1] 1789136257.196870 mapwrite 0x21d000 thru 0x221e05 (0x4e06 bytes) 000301[0] 1789136257.275513 mapwrite 0x247000 thru 0x249ff9 (0x2ffa bytes) 000302[0] 1789136257.366393 write_OD 0x4d0000 thru 0x4dcfff (0xd000 bytes) 000307[1] 1789136257.438186 mapread 0x4ec000 thru 0x4f33ac (0x73ad bytes) 000308[0] 1789136257.483668 write_OD 0x16b000 thru 0x179fff (0xf000 bytes) 000310[1] 1789136257.551789 write_OD 0x52b000 thru 0x539fff (0xf000 bytes) 000313[0] 1789136257.608297 read_OD 0x5ac000 thru 0x5b4fff (0x9000 bytes) 000314[0] 1789136257.618475 trunc from 0x740929 to 0x3568c3 (0x3ea065 bytes) 000316[1] 1789136257.688543 mapread 0x07b000 thru 0x08a8a4 (0xf8a5 bytes) 000318[0] 1789136257.780399 write_OD 0x625000 thru 0x625fff (0x1000 bytes) 000320[1] 1789136257.870688 mapread 0x4f9000 thru 0x5018c4 (0x88c5 bytes) 000322[1] 1789136257.942339 mapread 0x004000 thru 0x0087fb (0x47fc bytes) 000324[1] 1789136257.981414 write_OD 0x858000 thru 0x85afff (0x3000 bytes) 000326[0] 1789136258.051345 trunc from 0x85b000 to 0x173591 (0x6e7a6e bytes) 000327[1] 1789136258.090151 trunc from 0x173592 to 0x5898ec (0x41635b bytes) 000328[1] 1789136258.168133 mapwrite 0x1f0000 thru 0x1f5371 (0x5372 bytes) 000329[1] 1789136258.331546 write_OD 0x14b000 thru 0x14efff (0x4000 bytes) 000331[1] 1789136258.388036 trunc from 0x5898ed to 0x377167 (0x212785 bytes) 000333[0] 1789136258.424482 write_OD 0x18a000 thru 0x191fff (0x8000 bytes) 000335[0] 1789136258.489228 mapread 0x0d0000 thru 0x0dd4cc (0xd4cd bytes) 000336[0] 1789136258.523338 mapread 0x1a6000 thru 0x1a94ae (0x34af bytes) 000337[0] 1789136258.531522 write_OD 0x699000 thru 0x6a7fff (0xf000 bytes) 000338[1] 1789136258.611208 mapwrite 0x089000 thru 0x094927 (0xb928 bytes) 000341[0] 1789136258.823646 trunc from 0x6a8000 to 0x57334e (0x134cb1 bytes) 000343[0] 1789136258.931476 mapread 0x18f000 thru 0x190bcf (0x1bd0 bytes) 000347[0] 1789136259.040894 read_OD 0x286000 thru 0x28afff (0x5000 bytes) 000348[0] 1789136259.077232 read_OD 0x0b8000 thru 0x0b8fff (0x1000 bytes) 000349[1] 1789136259.094361 mapread 0x25b000 thru 0x2696d0 (0xe6d1 bytes) 000353[1] 1789136259.157695 trunc from 0x57334f to 0x42722f (0x14c11f bytes) 000354[1] 1789136259.233204 write_OD 0x692000 thru 0x6a0fff (0xf000 bytes) 000356[0] 1789136259.287954 mapread 0x0a0000 thru 0x0ab9ec (0xb9ed bytes) 000358[1] 1789136259.320229 trunc from 0x6a1000 to 0x37ec49 (0x3223b6 bytes) 000362[0] 1789136259.402141 mapwrite 0x37b000 thru 0x37f8e3 (0x48e4 bytes) 000366[0] 1789136259.591938 mapwrite 0x964000 thru 0x96b62f (0x7630 bytes) truncating to largest ever: 0x9ff79c 000367[0] 1789136259.711011 trunc from 0x96b630 to 0x9ff79b (0x9416c bytes) 000370[0] 1789136259.745204 mapread 0x4e0000 thru 0x4e8ed6 (0x8ed7 bytes) 000378[0] 1789136259.857200 write_OD 0x7b9000 thru 0x7bafff (0x2000 bytes) 000379[0] 1789136259.908476 mapread 0x520000 thru 0x52d873 (0xd874 bytes) 000381[1] 1789136259.934357 write_OD 0x708000 thru 0x716fff (0xf000 bytes) 000382[0] 1789136260.003275 mapread 0x026000 thru 0x026fb2 (0xfb3 bytes) 000383[1] 1789136260.013261 read_OD 0x907000 thru 0x910fff (0xa000 bytes) 000385[0] 1789136260.023447 read_OD 0x494000 thru 0x496fff (0x3000 bytes) 000391[1] 1789136260.031229 trunc from 0x9ff79c to 0x2f8d89 (0x706a12 bytes) 000396[1] 1789136260.114520 trunc from 0x2f8d8a to 0x96843c (0x66f6b3 bytes) 000397[1] 1789136260.159492 write_OD 0x754000 thru 0x75cfff (0x9000 bytes) 000398[1] 1789136260.232218 mapwrite 0x194000 thru 0x194726 (0x727 bytes) 000400[0] 1789136260.306470 read_OD 0x61d000 thru 0x61efff (0x2000 bytes) 000401[0] 1789136260.351581 write_OD 0x2ed000 thru 0x2fbfff (0xf000 bytes) 000402[0] 1789136260.425172 mapwrite 0x18c000 thru 0x1947be (0x87bf bytes) 000406[0] 1789136260.613786 trunc from 0x96843d to 0x7f647a (0x171fc2 bytes) 000407[0] 1789136260.648463 mapread 0x5f3000 thru 0x5f852b (0x552c bytes) 000408[0] 1789136260.726073 read_OD 0x393000 thru 0x39cfff (0xa000 bytes) 000410[0] 1789136260.762403 mapwrite 0x7c1000 thru 0x7cadc0 (0x9dc1 bytes) 000414[1] 1789136260.870564 mapread 0x009000 thru 0x00ac3e (0x1c3f bytes) 000417[0] 1789136260.901238 read_OD 0x4e7000 thru 0x4e8fff (0x2000 bytes) 000418[0] 1789136260.909426 write_OD 0x645000 thru 0x64bfff (0x7000 bytes) 000419[0] 1789136260.971937 read_OD 0x137000 thru 0x13cfff (0x6000 bytes) 000420[1] 1789136260.985165 write_OD 0x686000 thru 0x691fff (0xc000 bytes) 000422[1] 1789136261.041819 trunc from 0x7f647b to 0x5543d2 (0x2a20a8 bytes) 000423[1] 1789136261.084328 read_OD 0x2fd000 thru 0x30afff (0xe000 bytes) 000426[1] 1789136261.132606 write_OD 0x3dc000 thru 0x3e1fff (0x6000 bytes) 000430[1] 1789136261.193574 write_OD 0x610000 thru 0x61efff (0xf000 bytes) 000431[1] 1789136261.268228 read_OD 0x235000 thru 0x235fff (0x1000 bytes) 000433[0] 1789136261.277216 mapread 0x23f000 thru 0x24d240 (0xe241 bytes) 000434[1] 1789136261.303577 trunc from 0x61f000 to 0x2bb8b1 (0x36374e bytes) 000436[1] 1789136261.334160 trunc from 0x2bb8b2 to 0x01912d (0x2a2784 bytes) 000442[1] 1789136261.383461 mapread 0x017000 thru 0x018651 (0x1652 bytes) 000445[0] 1789136261.405991 mapwrite 0x839000 thru 0x844c84 (0xbc85 bytes) 000447[1] 1789136261.550584 mapread 0x6e9000 thru 0x6ed94b (0x494c bytes) 000449[1] 1789136261.599955 read_OD 0x703000 thru 0x707fff (0x5000 bytes) 000451[0] 1789136261.627301 write_OD 0x32d000 thru 0x330fff (0x4000 bytes) 000452[0] 1789136261.711820 read_OD 0x1bc000 thru 0x1c3fff (0x8000 bytes) 000456[0] 1789136261.732680 trunc from 0x844c85 to 0x83f134 (0x5b50 bytes) 000457[1] 1789136261.859446 mapread 0x4dd000 thru 0x4df110 (0x2111 bytes) 000458[0] 1789136261.927458 mapwrite 0x2e7000 thru 0x2f49b0 (0xd9b1 bytes) 000459[0] 1789136262.142910 trunc from 0x83f135 to 0x51b072 (0x3240c2 bytes) 000461[1] 1789136262.193926 read_OD 0x41f000 thru 0x423fff (0x5000 bytes) 000465[1] 1789136262.250085 write_OD 0x157000 thru 0x15cfff (0x6000 bytes) 000472[0] 1789136262.310788 trunc from 0x51b073 to 0x34c772 (0x1ce900 bytes) 000473[0] 1789136262.341372 write_OD 0x09c000 thru 0x0a8fff (0xd000 bytes) 000474[1] 1789136262.401397 mapwrite 0x77c000 thru 0x782688 (0x6689 bytes) 000475[0] 1789136262.506657 mapwrite 0x92d000 thru 0x932d4a (0x5d4b bytes) 000476[1] 1789136262.624195 trunc from 0x932d4b to 0x8c7181 (0x6bbc9 bytes) 000483[0] 1789136262.847014 mapwrite 0x5b2000 thru 0x5c1287 (0xf288 bytes) 000484[1] 1789136263.151567 mapread 0x401000 thru 0x4035c8 (0x25c9 bytes) 000485[1] 1789136263.258796 mapwrite 0x8db000 thru 0x8df42a (0x442b bytes) 000486[0] 1789136263.528354 mapwrite 0x95b000 thru 0x96ad47 (0xfd48 bytes) 000489[0] 1789136263.690645 write_OD 0x62e000 thru 0x63bfff (0xe000 bytes) 000492[1] 1789136263.851955 mapread 0x0b9000 thru 0x0c284e (0x984f bytes) 000493[0] 1789136263.889662 mapwrite 0x0f6000 thru 0x105541 (0xf542 bytes) 000496[1] 1789136264.099932 mapread 0x2ee000 thru 0x2f1b90 (0x3b91 bytes) 000499[0] 1789136264.168046 mapread 0x246000 thru 0x248185 (0x2186 bytes) All operations completed A-OK! PASS 16a (49s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 10:17:49 (1789136269) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.37805 s, 22.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.336636 s, 24.9 MB/s Chance of close/open is 1 in 50 Seed set to 6272 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x3afad1 000004[0] 1789136272.910750 trunc from 00000000 to 0x3afad0 (0x3afad1 bytes) 000005[1] 1789136272.969471 mapread 0x036b23 thru 0x038451 (0x192f bytes) truncating to largest ever: 0x6c4672 000006[1] 1789136273.059275 trunc from 0x3afad1 to 0x6c4671 (0x314ba1 bytes) 000013[1] 1789136273.153829 write 0x561bbf thru 0x5631e8 (0x162a bytes) 000016[1] 1789136273.181270 read 0x491b9a thru 0x493651 (0x1ab8 bytes) 000017[1] 1789136273.203979 mapread 0x17b342 thru 0x17c133 (0xdf2 bytes) 000020[1] 1789136273.231994 mapread 0x5d4bd1 thru 0x5d578a (0xbba bytes) 000024[0] 1789136273.298890 read 0x3e5c82 thru 0x3e6578 (0x8f7 bytes) 000025[1] 1789136273.335637 read 0x20c15e thru 0x20dce6 (0x1b89 bytes) 000026[1] 1789136273.345304 mapread 0x26700d thru 0x267cd8 (0xccc bytes) truncating to largest ever: 0x6d7c14 000028[1] 1789136273.382333 trunc from 0x6c4672 to 0x6d7c13 (0x135a2 bytes) 000030[0] 1789136273.484075 write 0x9e2808 thru 0x9e46b1 (0x1eaa bytes) 000031[0] 1789136273.516968 mapread 0x232630 thru 0x23428a (0x1c5b bytes) 000034[1] 1789136273.608694 trunc from 0x9e46b2 to 0x3a8706 (0x63bfab bytes) 000036[0] 1789136273.650574 read 0x37693a thru 0x378855 (0x1f1c bytes) 000038[0] 1789136273.716648 mapread 0x32a376 thru 0x32b1f7 (0xe82 bytes) truncating to largest ever: 0x9f62e5 000039[1] 1789136273.741057 trunc from 0x3a8707 to 0x9f62e4 (0x64dbde bytes) 000046[0] 1789136273.859204 mapread 0x9ce68a thru 0x9d0138 (0x1aaf bytes) 000049[0] 1789136273.962273 mapwrite 0x993a2d thru 0x994960 (0xf34 bytes) 000050[1] 1789136274.124807 mapwrite 0x3adc2e thru 0x3adf8f (0x362 bytes) 000051[0] 1789136274.220111 read 0x385b29 thru 0x3865fc (0xad4 bytes) 000052[0] 1789136274.256065 read 0x007070 thru 0x008706 (0x1697 bytes) 000055[0] 1789136274.288315 write 0x2173bc thru 0x217e32 (0xa77 bytes) 000061[0] 1789136274.435320 read 0x7cddbb thru 0x7cfbb7 (0x1dfd bytes) 000063[0] 1789136274.476465 mapread 0x2ff1c3 thru 0x2ff556 (0x394 bytes) 000064[0] 1789136274.494224 mapread 0x01853c thru 0x019753 (0x1218 bytes) 000066[1] 1789136274.523335 read 0x1db471 thru 0x1dcdf2 (0x1982 bytes) 000068[1] 1789136274.576661 mapread 0x7fb4b8 thru 0x7fc5a8 (0x10f1 bytes) 000070[0] 1789136274.589103 trunc from 0x9f62e5 to 0x37a35d (0x67bf87 bytes) 000072[1] 1789136274.703498 write 0x6a6c24 thru 0x6a8bca (0x1fa7 bytes) 000073[1] 1789136274.750139 trunc from 0x6a8bcb to 0x79db1c (0xf4f52 bytes) 000075[0] 1789136274.809689 mapwrite 0x2e0067 thru 0x2e0279 (0x213 bytes) 000076[1] 1789136274.953999 read 0x05d097 thru 0x05e349 (0x12b3 bytes) 000078[1] 1789136275.032702 write 0x132f77 thru 0x133ae4 (0xb6e bytes) 000079[1] 1789136275.090277 mapwrite 0x94688f thru 0x9473d8 (0xb4a bytes) 000082[0] 1789136275.248675 trunc from 0x9473d9 to 0x011e2a (0x9355ae bytes) 000083[0] 1789136275.333006 mapwrite 0x4470aa thru 0x448d58 (0x1caf bytes) 000084[0] 1789136275.417666 read 0x13bab5 thru 0x13c8fb (0xe47 bytes) 000086[1] 1789136275.448299 mapread 0x419a26 thru 0x41b6fd (0x1cd8 bytes) 000087[1] 1789136275.495187 trunc from 0x448d59 to 0x110e94 (0x337ec4 bytes) 000090[0] 1789136275.623024 mapread 0x02b487 thru 0x02ccdb (0x1855 bytes) 000091[0] 1789136275.658213 trunc from 0x110e95 to 0x936c90 (0x825dfc bytes) 000094[0] 1789136275.751073 mapwrite 0x038a9c thru 0x0397ea (0xd4f bytes) 000096[0] 1789136275.841482 write 0x9c1daa thru 0x9c3b71 (0x1dc8 bytes) 000097[0] 1789136275.842654 trunc from 0x9c3b72 to 0x1f6cf4 (0x7cce7d bytes) 000098[1] 1789136275.879301 mapread 0x0e1d9c thru 0x0e2ca5 (0xf0a bytes) 000099[0] 1789136275.937401 read 0x0761b2 thru 0x07657b (0x3ca bytes) 000100[0] 1789136276.005013 write 0x4f3f0b thru 0x4f4b81 (0xc77 bytes) 000102[0] 1789136276.025929 mapwrite 0x99cb75 thru 0x99d1c6 (0x652 bytes) 000103[0] 1789136276.275373 write 0x865d12 thru 0x866537 (0x826 bytes) 000104[1] 1789136276.303866 mapwrite 0x4773c2 thru 0x4790fd (0x1d3c bytes) 000108[0] 1789136276.443263 mapwrite 0x1a1560 thru 0x1a2554 (0xff5 bytes) 000109[0] 1789136276.563604 write 0x5f51b2 thru 0x5f6c3e (0x1a8d bytes) 000110[1] 1789136276.639052 mapwrite 0x9e4d08 thru 0x9e5eb4 (0x11ad bytes) 000112[1] 1789136276.880362 trunc from 0x9e5eb5 to 0x7d5eb7 (0x20fffd bytes) 000113[1] 1789136276.905015 mapwrite 0x52c60a thru 0x52cd54 (0x74b bytes) 000114[0] 1789136277.003309 write 0x9ac9d7 thru 0x9ae68d (0x1cb7 bytes) 000116[1] 1789136277.048294 mapread 0x460ae9 thru 0x460d33 (0x24b bytes) 000117[0] 1789136277.073692 trunc from 0x9ae68e to 0x9b39e1 (0x5354 bytes) 000119[0] 1789136277.112837 mapwrite 0x423978 thru 0x425510 (0x1b99 bytes) 000125[0] 1789136277.248794 write 0x327b86 thru 0x327f0e (0x389 bytes) 000126[0] 1789136277.255080 write 0x403935 thru 0x404424 (0xaf0 bytes) 000127[0] 1789136277.288177 read 0x7e120e thru 0x7e25a8 (0x139b bytes) 000128[1] 1789136277.314632 mapread 0x18ba35 thru 0x18c5e9 (0xbb5 bytes) 000130[0] 1789136277.355219 trunc from 0x9b39e2 to 0x65fa5b (0x353f86 bytes) 000131[0] 1789136277.423577 mapread 0x085ff4 thru 0x08771f (0x172c bytes) 000133[0] 1789136277.492237 mapread 0x1f91cb thru 0x1fa68a (0x14c0 bytes) 000135[1] 1789136277.540906 mapread 0x2aa30a thru 0x2ac08d (0x1d84 bytes) 000137[1] 1789136277.613434 mapwrite 0x5eb505 thru 0x5ec532 (0x102e bytes) 000138[0] 1789136277.716899 write 0x5132e3 thru 0x515033 (0x1d51 bytes) 000139[0] 1789136277.749177 trunc from 0x65fa5c to 0x0872cb (0x5d8790 bytes) 000140[1] 1789136277.818310 mapwrite 0x1366f3 thru 0x137fed (0x18fb bytes) 000142[0] 1789136277.984398 read 0x0de092 thru 0x0dfd78 (0x1ce7 bytes) 000144[0] 1789136278.046939 write 0x789338 thru 0x78b0a5 (0x1d6e bytes) 000146[1] 1789136278.081122 trunc from 0x78b0a6 to 0x4178d6 (0x3737cf bytes) 000147[1] 1789136278.143930 mapwrite 0x53c9b2 thru 0x53cef1 (0x540 bytes) 000148[0] 1789136278.298341 mapwrite 0x52b8a1 thru 0x52c735 (0xe95 bytes) 000149[0] 1789136278.478954 mapwrite 0x0cf83e thru 0x0d0945 (0x1108 bytes) 000150[1] 1789136278.724293 mapread 0x4c04e3 thru 0x4c0cf9 (0x817 bytes) 000151[0] 1789136278.811617 mapread 0x171e75 thru 0x172ad8 (0xc64 bytes) 000152[0] 1789136278.825410 read 0x3cfa98 thru 0x3d156b (0x1ad4 bytes) 000153[0] 1789136278.859360 mapwrite 0x73840a thru 0x73a36e (0x1f65 bytes) 000155[0] 1789136279.020671 write 0x38159f thru 0x38196a (0x3cc bytes) 000157[1] 1789136279.064139 write 0x372b5d thru 0x373571 (0xa15 bytes) 000158[1] 1789136279.129296 write 0x762b16 thru 0x763033 (0x51e bytes) 000160[0] 1789136279.142050 mapread 0x08f88b thru 0x090593 (0xd09 bytes) 000161[1] 1789136279.186104 write 0x8e9e38 thru 0x8eb51c (0x16e5 bytes) 000163[0] 1789136279.199666 mapwrite 0x2896c6 thru 0x28a96c (0x12a7 bytes) 000164[0] 1789136279.288576 read 0x45239f thru 0x452c2a (0x88c bytes) 000165[1] 1789136279.309256 mapread 0x7d5d54 thru 0x7d6a83 (0xd30 bytes) 000166[0] 1789136279.399327 write 0x00442a thru 0x005473 (0x104a bytes) 000168[1] 1789136279.444189 mapread 0x4a6e97 thru 0x4a8b3f (0x1ca9 bytes) 000171[0] 1789136279.490337 mapwrite 0x0b2395 thru 0x0b2daa (0xa16 bytes) 000172[1] 1789136279.553686 mapread 0x23d0f4 thru 0x23e9eb (0x18f8 bytes) 000173[1] 1789136279.618526 trunc from 0x8eb51d to 0x6d8285 (0x213297 bytes) 000174[0] 1789136279.699697 trunc from 0x6d8286 to 0x981e9c (0x2a9c17 bytes) 000175[1] 1789136279.761459 trunc from 0x981e9d to 0x07d4c7 (0x9049d5 bytes) 000176[1] 1789136279.826224 read 0x02deda thru 0x02fa7e (0x1ba5 bytes) 000177[1] 1789136279.877906 mapwrite 0x6cdc4b thru 0x6cf788 (0x1b3e bytes) 000184[1] 1789136280.256113 read 0x270bcc thru 0x271ff7 (0x142c bytes) 000191[1] 1789136280.397237 read 0x431e58 thru 0x432258 (0x401 bytes) 000193[0] 1789136280.421065 read 0x2a8ae9 thru 0x2aa4e4 (0x19fc bytes) 000194[0] 1789136280.460913 mapwrite 0x65f22f thru 0x660aa2 (0x1874 bytes) 000196[1] 1789136280.549926 trunc from 0x6cf789 to 0x98a3da (0x2bac52 bytes) 000197[0] 1789136280.601472 mapread 0x115e22 thru 0x1173d3 (0x15b2 bytes) 000198[1] 1789136280.686764 trunc from 0x98a3db to 0x841692 (0x148d48 bytes) 000199[0] 1789136280.806978 mapread 0x09ae0f thru 0x09be76 (0x1068 bytes) 000202[1] 1789136280.899870 trunc from 0x841693 to 0x20c258 (0x63543a bytes) 000209[0] 1789136281.063682 write 0x01d427 thru 0x01f10c (0x1ce6 bytes) 000212[1] 1789136281.146684 mapread 0x1c60e8 thru 0x1c627b (0x194 bytes) 000215[0] 1789136281.230835 mapwrite 0x9f8d30 thru 0x9f9f7c (0x124d bytes) 000218[0] 1789136281.393440 trunc from 0x9f9f7d to 0x85debd (0x19c0bf bytes) 000221[1] 1789136281.442187 mapwrite 0x1f743a thru 0x1f92a8 (0x1e6f bytes) 000223[0] 1789136281.778864 read 0x354098 thru 0x355334 (0x129d bytes) 000232[0] 1789136281.851557 write 0x00aeb7 thru 0x00b19a (0x2e4 bytes) 000233[1] 1789136281.965744 mapread 0x03ec6e thru 0x040c1c (0x1faf bytes) 000234[0] 1789136282.079968 mapread 0x364343 thru 0x364a9b (0x759 bytes) 000235[0] 1789136282.093059 mapwrite 0x839a43 thru 0x83a1e6 (0x7a4 bytes) 000236[0] 1789136282.193619 write 0x73aa12 thru 0x73b99a (0xf89 bytes) 000237[0] 1789136282.203442 read 0x0add05 thru 0x0aee9d (0x1199 bytes) 000238[0] 1789136282.208275 trunc from 0x85debe to 0x404eb0 (0x45900d bytes) 000240[0] 1789136282.280217 write 0x6d6eda thru 0x6d6faa (0xd1 bytes) 000241[0] 1789136282.300849 mapwrite 0x802ac8 thru 0x803c44 (0x117d bytes) 000246[0] 1789136282.491978 read 0x4ac0b9 thru 0x4ace21 (0xd69 bytes) 000247[0] 1789136282.506934 write 0x4d43d0 thru 0x4d4b16 (0x747 bytes) 000248[1] 1789136282.528684 trunc from 0x803c45 to 0x310610 (0x4f3634 bytes) 000249[0] 1789136282.599206 mapread 0x2d2fd5 thru 0x2d3b24 (0xb50 bytes) 000254[1] 1789136282.689055 read 0x287c8a thru 0x28864f (0x9c6 bytes) 000256[0] 1789136282.827850 trunc from 0x310611 to 0x1d86a1 (0x137f6f bytes) 000257[1] 1789136282.918014 mapread 0x085c5c thru 0x086c6f (0x1014 bytes) 000258[0] 1789136282.961165 mapwrite 0x7f832e thru 0x7f875e (0x431 bytes) 000259[0] 1789136283.062972 trunc from 0x7f875f to 0x965ee1 (0x16d783 bytes) 000262[0] 1789136283.133689 trunc from 0x965ee2 to 0x04c531 (0x9199b0 bytes) 000263[0] 1789136283.165941 write 0x02bc07 thru 0x02c279 (0x673 bytes) 000265[1] 1789136283.227931 trunc from 0x04c532 to 0x888922 (0x83c3f1 bytes) 000270[1] 1789136283.321911 read 0x43e083 thru 0x43f10a (0x1088 bytes) 000272[0] 1789136283.330019 write 0x0cfb99 thru 0x0d033e (0x7a6 bytes) 000273[0] 1789136283.365097 mapread 0x2e49f0 thru 0x2e6048 (0x1659 bytes) 000274[1] 1789136283.407205 write 0x7d66bc thru 0x7d81e6 (0x1b2b bytes) 000278[0] 1789136283.510631 mapwrite 0x044ec7 thru 0x046483 (0x15bd bytes) 000279[1] 1789136283.627166 read 0x64d7ed thru 0x64eccb (0x14df bytes) 000283[0] 1789136283.699771 write 0x3d6cff thru 0x3d769e (0x9a0 bytes) 000284[1] 1789136283.729100 trunc from 0x888923 to 0x56c0f3 (0x31c82f bytes) 000286[1] 1789136283.773635 write 0x86fd33 thru 0x8707fb (0xac9 bytes) 000288[0] 1789136283.805583 mapwrite 0x2bc567 thru 0x2bd5a9 (0x1043 bytes) 000291[1] 1789136284.104692 mapread 0x5e1dbf thru 0x5e1e54 (0x96 bytes) 000294[1] 1789136284.207872 mapwrite 0x5e3481 thru 0x5e4d20 (0x18a0 bytes) 000295[1] 1789136284.317935 trunc from 0x8707fc to 0x2ee6bb (0x582140 bytes) 000301[1] 1789136284.411582 mapread 0x004a99 thru 0x00585c (0xdc4 bytes) 000302[0] 1789136284.443233 mapread 0x0d419b thru 0x0d4f25 (0xd8b bytes) 000304[0] 1789136284.471866 read 0x0eca58 thru 0x0ed670 (0xc19 bytes) 000305[0] 1789136284.530231 mapread 0x1b1cc3 thru 0x1b34d1 (0x180f bytes) 000307[1] 1789136284.563969 mapwrite 0x89e90e thru 0x89fb25 (0x1218 bytes) 000310[0] 1789136284.678270 write 0x8f7957 thru 0x8f8a91 (0x113b bytes) 000311[0] 1789136284.708610 read 0x794800 thru 0x795ed2 (0x16d3 bytes) 000312[1] 1789136284.745948 mapread 0x4c69d9 thru 0x4c7bf9 (0x1221 bytes) 000314[0] 1789136284.777941 mapwrite 0x74cfc7 thru 0x74d668 (0x6a2 bytes) 000316[0] 1789136284.873988 trunc from 0x8f8a92 to 0x159ec2 (0x79ebcf bytes) 000317[1] 1789136284.902251 read 0x104417 thru 0x105c3b (0x1825 bytes) 000320[0] 1789136284.929458 mapread 0x157544 thru 0x1588ea (0x13a7 bytes) 000321[1] 1789136284.939658 read 0x10cb61 thru 0x10d98d (0xe2d bytes) 000328[1] 1789136285.012892 trunc from 0x159ec3 to 0x8b40f6 (0x75a234 bytes) 000329[0] 1789136285.091409 mapwrite 0x338415 thru 0x338b31 (0x71d bytes) 000330[0] 1789136285.170429 read 0x38dc74 thru 0x38eee6 (0x1273 bytes) 000331[1] 1789136285.193874 mapread 0x0e40ed thru 0x0e44c0 (0x3d4 bytes) 000332[1] 1789136285.245097 write 0x5ef0b5 thru 0x5efd0d (0xc59 bytes) 000333[1] 1789136285.326833 trunc from 0x8b40f7 to 0x109520 (0x7aabd6 bytes) 000334[0] 1789136285.380456 mapwrite 0x069b26 thru 0x069eaf (0x38a bytes) 000335[0] 1789136285.522447 mapwrite 0x8e7669 thru 0x8e76e2 (0x7a bytes) 000336[1] 1789136285.697256 mapwrite 0x19cf1e thru 0x19ecbd (0x1da0 bytes) 000337[0] 1789136285.841825 trunc from 0x8e76e3 to 0x43e1be (0x4a9524 bytes) 000338[0] 1789136285.904522 trunc from 0x43e1bf to 0x8fcc4a (0x4bea8c bytes) 000341[1] 1789136285.985011 write 0x11a037 thru 0x11a86d (0x837 bytes) 000343[0] 1789136286.036800 read 0x79f286 thru 0x79f5b2 (0x32d bytes) 000347[0] 1789136286.118681 write 0x087fb7 thru 0x088bbf (0xc09 bytes) 000349[1] 1789136286.178928 write 0x138e63 thru 0x13aa6d (0x1c0b bytes) 000351[0] 1789136286.239429 trunc from 0x8fcc4b to 0x3b12d6 (0x54b974 bytes) 000353[0] 1789136286.336593 read 0x02cf39 thru 0x02d7ec (0x8b4 bytes) 000355[1] 1789136286.444193 write 0x0c08e6 thru 0x0c1ca6 (0x13c1 bytes) 000357[0] 1789136286.522074 mapwrite 0x971325 thru 0x9732c1 (0x1f9d bytes) 000358[1] 1789136286.741089 write 0x1bab72 thru 0x1bb0d1 (0x560 bytes) 000359[0] 1789136286.842895 mapwrite 0x8ca322 thru 0x8ca6ac (0x38b bytes) 000360[0] 1789136287.176209 trunc from 0x9732c2 to 0x48925c (0x4ea065 bytes) 000364[1] 1789136287.216282 write 0x36177d thru 0x362cc2 (0x1546 bytes) 000367[0] 1789136287.306600 mapwrite 0x2a89d1 thru 0x2aa20a (0x183a bytes) 000368[0] 1789136287.429674 mapwrite 0x2a6728 thru 0x2a7251 (0xb2a bytes) 000370[1] 1789136287.641821 mapread 0x1eafa3 thru 0x1ecea9 (0x1f07 bytes) 000376[0] 1789136287.796972 write 0x41fc56 thru 0x421157 (0x1502 bytes) 000377[1] 1789136287.838390 mapwrite 0x868efd thru 0x869d45 (0xe49 bytes) 000380[0] 1789136288.033657 write 0x1cb9fb thru 0x1cc14a (0x750 bytes) 000386[1] 1789136288.156043 mapwrite 0x3fe171 thru 0x400086 (0x1f16 bytes) 000387[1] 1789136288.289202 mapwrite 0x181aa5 thru 0x181e4b (0x3a7 bytes) 000388[1] 1789136288.352043 read 0x681f70 thru 0x68265f (0x6f0 bytes) 000389[0] 1789136288.358325 mapread 0x016b65 thru 0x017ebb (0x1357 bytes) 000391[0] 1789136288.397003 mapwrite 0x6a66b4 thru 0x6a6df6 (0x743 bytes) 000392[0] 1789136288.463451 read 0x65e05e thru 0x65fe8d (0x1e30 bytes) 000394[0] 1789136288.494074 read 0x2b7543 thru 0x2b8cc1 (0x177f bytes) 000395[1] 1789136288.516305 read 0x7cec19 thru 0x7d0181 (0x1569 bytes) 000397[0] 1789136288.538805 trunc from 0x869d46 to 0x9200e6 (0xb63a1 bytes) 000398[0] 1789136288.608096 write 0x6201d6 thru 0x621157 (0xf82 bytes) 000399[1] 1789136288.627449 read 0x618ea4 thru 0x61a581 (0x16de bytes) 000400[0] 1789136288.649545 write 0x78c63a thru 0x78e147 (0x1b0e bytes) 000402[1] 1789136288.687166 trunc from 0x9200e7 to 0x366294 (0x5b9e52 bytes) 000403[1] 1789136288.782962 read 0x19e961 thru 0x19eee6 (0x586 bytes) 000406[1] 1789136288.897294 write 0x4202be thru 0x421b14 (0x1857 bytes) 000412[0] 1789136288.965915 write 0x944fed thru 0x9456b8 (0x6cc bytes) 000413[0] 1789136289.010396 trunc from 0x9456b9 to 0x828fab (0x11c70d bytes) 000415[0] 1789136289.084115 trunc from 0x828fac to 0x09d925 (0x78b686 bytes) 000416[1] 1789136289.106574 mapread 0x06bdd3 thru 0x06dadc (0x1d0a bytes) 000420[0] 1789136289.245667 mapwrite 0x94476d thru 0x945a8f (0x1323 bytes) 000422[1] 1789136289.421662 mapread 0x16edd6 thru 0x170921 (0x1b4c bytes) 000423[1] 1789136289.499139 write 0x00d804 thru 0x00e5d3 (0xdd0 bytes) 000424[1] 1789136289.536479 trunc from 0x945a90 to 0x8da185 (0x6b90a bytes) 000425[0] 1789136289.590703 trunc from 0x8da186 to 0x359d46 (0x58043f bytes) 000426[1] 1789136289.634913 mapread 0x275fcd thru 0x2770e8 (0x111c bytes) 000427[1] 1789136289.674061 trunc from 0x359d47 to 0x6f4656 (0x39a910 bytes) 000428[1] 1789136289.736316 mapwrite 0x68a311 thru 0x68beba (0x1baa bytes) 000431[0] 1789136289.858796 mapread 0x0f7404 thru 0x0f908b (0x1c88 bytes) 000433[1] 1789136289.920609 read 0x2010a0 thru 0x201166 (0xc7 bytes) 000435[1] 1789136289.955777 trunc from 0x6f4657 to 0x3c1dda (0x33287c bytes) 000437[1] 1789136290.015278 write 0x2f20b3 thru 0x2f220e (0x15c bytes) 000438[0] 1789136290.053220 read 0x2ae7ef thru 0x2aed3e (0x550 bytes) 000439[1] 1789136290.177119 mapwrite 0x3f9517 thru 0x3fa89a (0x1384 bytes) 000440[1] 1789136290.300174 mapread 0x19a7ed thru 0x19c371 (0x1b85 bytes) 000441[0] 1789136290.335347 read 0x1e1767 thru 0x1e1c5f (0x4f9 bytes) 000442[1] 1789136290.441094 read 0x0e4773 thru 0x0e5e9e (0x172c bytes) 000443[0] 1789136290.526943 write 0x4ebbbd thru 0x4ecc74 (0x10b8 bytes) 000444[1] 1789136290.551742 mapwrite 0x2c6bc4 thru 0x2c7aaf (0xeec bytes) 000445[0] 1789136290.727896 mapwrite 0x22a384 thru 0x22bddd (0x1a5a bytes) 000446[0] 1789136290.829065 mapread 0x37e295 thru 0x37ef27 (0xc93 bytes) 000447[0] 1789136290.835019 trunc from 0x4ecc75 to 0x49aa51 (0x52223 bytes) 000448[1] 1789136290.857608 mapread 0x09235f thru 0x0935d5 (0x1277 bytes) 000450[1] 1789136290.932664 mapread 0x37a0cf thru 0x37a756 (0x688 bytes) 000451[1] 1789136290.945395 mapwrite 0x1afa28 thru 0x1b1499 (0x1a72 bytes) 000452[0] 1789136291.051567 read 0x35b6ed thru 0x35b6ff (0x13 bytes) 000454[1] 1789136291.087446 trunc from 0x49aa52 to 0x3070cf (0x193982 bytes) 000455[1] 1789136291.138133 mapwrite 0x0fe5c3 thru 0x0fffa1 (0x19df bytes) 000456[1] 1789136291.215339 trunc from 0x3070d0 to 0x48478f (0x17d6c0 bytes) 000457[1] 1789136291.289659 trunc from 0x484790 to 0x333cb7 (0x150ad8 bytes) 000462[0] 1789136291.354846 read 0x092cbe thru 0x0931c2 (0x505 bytes) 000463[1] 1789136291.448936 read 0x023013 thru 0x023bfc (0xbea bytes) 000467[1] 1789136291.505473 mapwrite 0x1731e5 thru 0x174d68 (0x1b84 bytes) 000469[0] 1789136291.661863 trunc from 0x333cb8 to 0x0ccd21 (0x266f96 bytes) 000473[0] 1789136291.851540 mapread 0x0c381b thru 0x0c49f4 (0x11da bytes) 000474[0] 1789136291.878024 read 0x053384 thru 0x05458f (0x120c bytes) 000476[0] 1789136291.946145 write 0x14b91d thru 0x14bbb1 (0x295 bytes) 000477[0] 1789136291.947129 write 0x08fc70 thru 0x090487 (0x818 bytes) 000478[1] 1789136291.947787 read 0x0343f5 thru 0x036235 (0x1e41 bytes) 000479[0] 1789136292.028914 mapwrite 0x19e089 thru 0x19f0d4 (0x104c bytes) 000480[0] 1789136292.174438 mapwrite 0x6e6088 thru 0x6e6782 (0x6fb bytes) 000484[1] 1789136292.442269 write 0x0d5ac3 thru 0x0d6aff (0x103d bytes) 000486[1] 1789136292.537342 trunc from 0x6e6783 to 0x174e24 (0x57195e bytes) 000487[1] 1789136292.573288 mapwrite 0x9fe0f5 thru 0x9ffd37 (0x1c43 bytes) 000493[0] 1789136292.851229 mapread 0x19c40b thru 0x19ce0e (0xa04 bytes) 000500[1] 1789136292.951546 trunc from 0x9ffd38 to 0x925037 (0xdad00 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 10:18:19 (1789136299) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 10:18:23 (1789136303) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.375627 s, 22.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.347403 s, 24.1 MB/s ldlm.namespaces.MGC192.168.204.132@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa091d940c800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa091e040e800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa091e040e800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa091e040e800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.617 s, 9.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.09894 s, 33.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.36987 s, 44.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19975 s, 20.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.14032 s, 14.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.59305 s, 29.2 MB/s PASS 16d (46s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 10:19:09 (1789136349) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.383616 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157355 s, 6.7 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 10:19:16 (1789136356) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 10:19:44 (1789136384) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 10:20:14 (1789136414) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 10:20:22 (1789136422) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 10:20:30 (1789136430) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44007 s, 10.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.06115 s, 24.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54158 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.70143 s, 15.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44118 s, 10.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0197 s, 13.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.67796 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20023 s, 21.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.59502 s, 16.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.57151 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.78958 s, 6.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26685 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50541 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.80133 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03892 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.73876 s, 7.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.90829 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.0517 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.16304 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (40s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 10:21:10 (1789136470) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 11 10:21:56 EDT 2026 + fsx 2 done: rc=0 Fri Sep 11 10:21:56 EDT 2026 + fsx 3 done: rc=0 Fri Sep 11 10:21:56 EDT 2026 + fsx 4 done: rc=0 Fri Sep 11 10:21:56 EDT 2026 + fsx 5 done: rc=0 Fri Sep 11 10:21:56 EDT 2026 PASS 16k (56s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 10:22:06 (1789136526) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 10:22:17 (1789136537) mmap test1: basic mmap operation (PASS, 0.15078s) mmap test2: MAP_PRIVATE not write back (PASS, 0.14032s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.565s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6905s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.12977s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18864s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3823s) mmap test11: multi-thread shared mmap (PASS, 1.4391s) mmap test10: mtime not change for readonly mmap access (PASS, 1.097s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 10:22:49 (1789136569) oleg432-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 10:22:54 (1789136574) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 10:23:01 (1789136581) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 10:23:07 (1789136587) now is 1789136589 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 new atime is 1789136651 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 10:24:17 (1789136657) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 403 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 403 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117267 403 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 403 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117267 403 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117267 403 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 10:24:24 (1789136664) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 10:24:32 (1789136672) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 10:24:39 (1789136679) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 10:24:42 (1789136682) utime: good mknod times 1789136682 <= 1789136683 <= 1789136683 for /mnt/lustre/f26a utime: good mknod times 1789136682 <= 1789136683 <= 1789136683 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 10:24:49 (1789136689) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 10:24:57 (1789136697) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 10:24:59.000000000 -0400 Birth: 2026-09-11 10:25:00.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 10:24:59.000000000 -0400 Birth: 2026-09-11 10:25:00.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 10:25:07 (1789136707) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 10:25:16 (1789136716) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 10:25:24 (1789136724) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.132@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 10:25:46 (1789136746) -rw-r--r-- 1 root root 0 Sep 11 10:25 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 10:25:55 (1789136755) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43337 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 10:26:02 (1789136762) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 10:26:10 (1789136770) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 10:26:13 (1789136773) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42748 s, 36.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0924 s, 48.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.70416 s, 30.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.87047 s, 28.0 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06415 s, 25.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.29266 s, 22.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44704 s, 36.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47805 s, 35.5 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 10:26:46 (1789136806) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 10:26:50 (1789136810) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 10:26:54 (1789136814) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 10:26:58 (1789136818) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 10:27:01 (1789136821) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa091d940c800.ost_server_uuid in IDLE state after 0 sec PASS 34 (141s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 10:29:22 (1789136962) fail_loc=0x80000317 Race attempt 0 Wait for 50640 50660 for 60 sec... - open/close 779 (time 1789136980.72 total 10.01 last 77.81) - open/close 1413 (time 1789136990.73 total 20.02 last 63.34) - open/close 2203 (time 1789137000.74 total 30.03 last 78.93) - open/close 3031 (time 1789137010.75 total 40.05 last 82.66) - open/close 3726 (time 1789137020.76 total 50.06 last 69.44) total: 4000 open/close in 53.43 seconds: 74.86 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.132@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.204.132@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 10:30:36 (1789137036) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30263 s, 40.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27775 s, 41.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37165 s, 38.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44795 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56465 s, 33.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34952 s, 38.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54835 s, 33.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50514 s, 34.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48744 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.1176 s, 46.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29177 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7443456) after(7494656) PASS 36 (297s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 10:35:32 (1789137332) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 - create 1403 (time 1789137346.87 total 10.01 last 140.22) - create 2682 (time 1789137356.87 total 20.01 last 127.86) - create 4155 (time 1789137366.88 total 30.02 last 147.17) - create 5411 (time 1789137376.88 total 40.02 last 125.58) - create 6581 (time 1789137386.89 total 50.03 last 116.91) - create 7797 (time 1789137396.90 total 60.04 last 121.49) - create 8933 (time 1789137406.90 total 70.04 last 113.55) total: 10000 create in 79.82 seconds: 125.29 ops/second PASS 37 (94s) == sanityn test 39a: file mtime does not change after rename ========================================================== 10:37:07 (1789137427) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 10:37:15 (1789137435) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 10:37:23 (1789137443) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 10:37:31 (1789137451) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 10:37:38 (1789137458) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 10:37:55 (1789137475) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 10:38:15 (1789137495) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 10:38:35 (1789137515) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 10:38:52 (1789137532) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 10:39:05 (1789137545) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 10:39:19 (1789137559) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 10:39:33 (1789137573) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 10:39:47 (1789137587) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 10:40:00 (1789137600) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 10:40:15 (1789137615) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 10:40:29 (1789137629) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 10:40:42 (1789137642) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1078s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:58:40 (1789138720) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:58:53 (1789138733) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:59:06 (1789138746) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (12s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:59:18 (1789138758) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:59:32 (1789138772) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:59:46 (1789138786) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 11:00:00 (1789138800) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 11:00:15 (1789138815) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 11:00:29 (1789138829) PASS 43a (72s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 11:01:41 (1789138901) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 11:01:55 (1789138915) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 11:02:11 (1789138931) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 11:02:26 (1789138946) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 11:02:40 (1789138960) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 11:02:54 (1789138974) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 11:03:08 (1789138988) Conflict PASS 43h (12s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 11:03:20 (1789139000) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 11:03:24 (1789139004) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (118s) == sanityn test 43k: unlink vs create ==================== 11:05:22 (1789139122) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1243s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 11:26:06 (1789140366) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 11 11:26:14 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 11:26:19 (1789140379) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 11:26:33 (1789140393) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 11:26:47 (1789140407) Conflict PASS 44d (17s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 11:27:04 (1789140424) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 11:27:19 (1789140439) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 11:27:35 (1789140455) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 11:27:48 (1789140468) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 11:28:02 (1789140482) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 11:28:06 (1789140486) PASS 45a (146s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 11:30:32 (1789140632) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 11:30:45 (1789140645) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 11:31:01 (1789140661) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 11:31:17 (1789140677) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 11:31:34 (1789140694) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (17s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 11:31:51 (1789140711) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 11:32:07 (1789140727) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 11:32:21 (1789140741) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 11:32:24 (1789140744) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1269s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 11:53:33 (1789142013) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 11:53:49 (1789142029) open: File exists Conflict PASS 46b (16s) == sanityn test 46c: pdirops: link vs link ======================================================================== 11:54:05 (1789142045) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 11:54:20 (1789142060) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 11:54:33 (1789142073) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 11:54:46 (1789142086) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 11:54:59 (1789142099) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 11:55:12 (1789142112) Conflict PASS 46h (12s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 11:55:24 (1789142124) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 11:55:27 (1789142127) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 11:55:31 (1789142131) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 11:55:33 (1789142133) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 11:55:37 (1789142137) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 11:55:40 (1789142140) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 11:55:43 (1789142143) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 11:55:45 (1789142145) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 11:55:48 (1789142148) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0918604 s, 111 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 11:56:00 (1789142160) 0+1 records in 0+1 records out 158 bytes copied, 0.0418485 s, 3.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218616) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 11:56:09 (1789142169) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0159357 s, 64.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786628 s, 13.3 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 11:56:29 (1789142189) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0282541 s, 36.2 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 11:56:41 (1789142201) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120194 s, 8.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 11:56:49 (1789142209) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 11:56:57 (1789142217) llite.lustre-ffffa091d940c800.xattr_cache=1 llite.lustre-ffffa091e040e800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:56:59.000000000 -0400 Modify: 2026-09-11 11:56:59.000000000 -0400 Change: 2026-09-11 11:56:59.000000000 -0400 Birth: 2026-09-11 11:56:59.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:57:05.000000000 -0400 Modify: 2026-09-11 11:57:06.000000000 -0400 Change: 2026-09-11 11:57:06.000000000 -0400 Birth: 2026-09-11 11:57:06.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:57:11.000000000 -0400 Modify: 2026-09-11 11:57:11.000000000 -0400 Change: 2026-09-11 11:57:11.000000000 -0400 Birth: 2026-09-11 11:57:12.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:57:11.000000000 -0400 Modify: 2026-09-11 11:57:13.000000000 -0400 Change: 2026-09-11 11:57:13.000000000 -0400 Birth: 2026-09-11 11:57:12.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:57:18.000000000 -0400 Modify: 2026-09-11 11:57:19.000000000 -0400 Change: 2026-09-11 11:57:19.000000000 -0400 Birth: 2026-09-11 11:57:19.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 11:57:18.000000000 -0400 Modify: 2026-09-11 11:57:19.000000000 -0400 Change: 2026-09-11 11:57:19.000000000 -0400 Birth: 2026-09-11 11:57:19.000000000 -0400 PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 11:57:29 (1789142249) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 11:57:42 (1789142262) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 11:57:56 (1789142276) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 11:58:16 (1789142296) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 11:58:32 (1789142312) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 11:58:35 (1789142315) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 11:58:55 (1789142335) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 11:59:17 (1789142357) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789142360 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.670759 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (100s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 12:00:57 (1789142457) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789142466 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.22074 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (116s) == sanityn test 60: Verify data_version behaviour ======== 12:02:53 (1789142573) PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 12:03:00 (1789142580) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 12:03:07 (1789142587) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 12:03:15 (1789142595) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0180836 s, 3.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 12:03:19 (1789142599) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0347111 s, 1.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 12:03:23 (1789142603) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 12:03:26 (1789142606) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0236166 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 12:03:29 (1789142609) llite.lustre-ffffa091d940c800.xattr_cache=1 llite.lustre-ffffa091e040e800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 12:03:35 (1789142615) llite.lustre-ffffa091d940c800.xattr_cache=1 llite.lustre-ffffa091e040e800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 12:03:41 (1789142621) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0178847 s, 57.3 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0262993 s, 38.9 kB/s 231441: get lock1 231440: get lock2 231441: done 231441: get lock2 231440: done 231440: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231440: cannot get lock1: Resource deadlock avoided 231441: done 231441: exit rc=0 231440: done 231440: exit rc=0 PASS 74 (10s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 12:03:51 (1789142631) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0709114 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429202 s, 24.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232022 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 12:04:02 (1789142642) opening files: ........ closing files: ........ PASS 76 (69s) == sanityn test 77a: check FIFO NRS policy =============== 12:05:11 (1789142711) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.460771 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427845 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696938 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173014 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153958 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250659 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214236 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270062 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246638 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273648 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31293 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267897 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319504 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23713 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271866 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286305 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364032 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466081 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378555 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429417 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246476 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504957 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905065 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932897 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439518 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586626 s, 17.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695635 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837472 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292216 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624475 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193889 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654841 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166044 s, 63.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 12:05:19 (1789142719) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434743 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448872 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834362 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964677 s, 10.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120202 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15596 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202358 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220749 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197802 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206765 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248373 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240072 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230716 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241216 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244655 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264331 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312777 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416439 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387146 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626279 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324824 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473243 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399198 s, 26.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258196 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718694 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592618 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397269 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320717 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585336 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067532 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557091 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645604 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546814 s, 192 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520369 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083387 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787201 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151768 s, 6.9 MB/sldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239587 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256115 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260822 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212418 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244361 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200128 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209949 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224309 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243751 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298255 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228711 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259546 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301535 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595314 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171604 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431853 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274349 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459137 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227776 s, 460 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107028 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892611 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535287 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729907 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445028 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400736 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04253 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421821 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353288 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380986 s, 27.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 12:05:31 (1789142731) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.352184 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654261 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741161 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121771 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781386 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13142 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119706 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107877 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903673 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127015 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199638 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181849 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195975 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21155 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246359 s, 4.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225395 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257759 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673944 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389078 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484268 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513205 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831269 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443205 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538556 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731312 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681876 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610156 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675691 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077105 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792953 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693158 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115916 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909766 s, 11.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.433341 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546848 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796133 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607132 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732035 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137548 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104818 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132155 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139346 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227264 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167847 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227125 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202251 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206501 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22236 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318438 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394565 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118561 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119815 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163799 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151904 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566385 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173949 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126996 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127286 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127069 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157058 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140207 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865734 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145687 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151812 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203026 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979015 s, 10.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441911 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668091 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683985 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159053 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169716 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170591 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218884 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267289 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238047 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251267 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233319 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253818 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299035 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282501 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306825 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29523 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353462 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591308 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472243 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364354 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325161 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17268 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206136 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137239 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112403 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129283 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804631 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943682 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159737 s, 65.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881687 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114893 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343263 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394201 s, 26.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (14s) == sanityn test 77d: check TRR nrs policy ================ 12:05:45 (1789142745) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.491709 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085373 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109845 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148192 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982828 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164656 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14198 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175758 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187982 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141033 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156315 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249649 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278332 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309199 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314604 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305719 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336141 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409151 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561983 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703521 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806789 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848329 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967692 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975564 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893165 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976586 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937509 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10066 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119571 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14459 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146555 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116382 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137144 s, 7.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.591061 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328299 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155662 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113992 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169548 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242799 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251829 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263587 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27849 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326992 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243933 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26288 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280812 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304957 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333455 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360731 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330018 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127339 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706055 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575548 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772577 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673728 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724718 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854369 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783141 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10659 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852325 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876202 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166852 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986024 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115266 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105814 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172309 s, 6.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445491 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775821 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162508 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14772 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124431 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145168 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256596 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196126 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269562 s, 3.9 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175672 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295874 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479245 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432206 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42936 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522125 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455004 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.604236 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843289 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225081 s, 466 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00709933 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120272 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119921 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319847 s, 328 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165537 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129059 s, 81.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237243 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204552 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10599 s, 9.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197536 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844004 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500565 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417981 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412652 s, 25.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 12:06:01 (1789142761) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.32@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.328222 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06307 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898567 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651743 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130416 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115349 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155268 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161747 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248934 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17221 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171546 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20868 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230725 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210903 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36616 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373623 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416999 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024622 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590924 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709631 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230517 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081373 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773729 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120124 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837332 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871063 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906241 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455329 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052966 s, 19.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104368 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622487 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639205 s, 16.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120313 s, 8.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.544353 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178716 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140205 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206076 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142594 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246282 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192431 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301248 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156625 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245331 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10279 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22795 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189869 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320756 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305984 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272939 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402492 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0087482 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782504 s, 134 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159166 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436381 s, 240 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171626 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088345 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810793 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478496 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582404 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540853 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040356 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108306 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348057 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331013 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277284 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320063 s, 32.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520576 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120319 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14079 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104849 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202681 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826104 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287376 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180072 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321851 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260616 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283246 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360215 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318458 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21217 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350003 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249077 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183457 s, 57.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394296 s, 26.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715046 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304651 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233338 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253672 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288407 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866373 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328943 s, 31.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375207 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392748 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379004 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959469 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433092 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440297 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714904 s, 14.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5425 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874413 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10103 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103144 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107267 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125639 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247599 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279949 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209482 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305045 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372221 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205528 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381534 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38989 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297736 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376596 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382332 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946121 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109789 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430798 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465397 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857449 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917427 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389339 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417926 s, 251 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684243 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560594 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253513 s, 414 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158411 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249967 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736696 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120508 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882335 s, 11.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 12:06:25 (1789142785) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428266 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541515 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058563 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716445 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101874 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116786 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197226 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167901 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11979 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16744 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188864 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252678 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237037 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304025 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321689 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354689 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321896 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106627 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806167 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095343 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00898515 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511037 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116218 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149581 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118467 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154763 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138095 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143205 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104139 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14674 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123621 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167333 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1643 s, 6.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.366235 s, 45.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624789 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911662 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270745 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203547 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245982 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251254 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305021 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267567 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31317 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348559 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349326 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291178 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307773 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261871 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2672 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388981 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110642 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252782 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013175 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195699 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828178 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402786 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403967 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700419 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137706 s, 76.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00734468 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010078 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749749 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427765 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209622 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177613 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401206 s, 26.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.487427 s, 34.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591773 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116171 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101121 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104557 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044919 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125672 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118103 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111556 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161941 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265363 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186814 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258746 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175762 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166316 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307083 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252069 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661379 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678668 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365764 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427306 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03873 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669931 s, 15.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444503 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433448 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445847 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058572 s, 17.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617604 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866649 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549364 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649994 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705931 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703848 s, 14.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.45282 s, 37.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832945 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11698 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142183 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127215 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145452 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124316 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206677 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231951 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126803 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26084 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270071 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223931 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118222 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206321 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304261 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23006 s, 4.6 MB/s execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595571 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602085 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813403 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790181 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657405 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670831 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765808 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965759 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852654 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874614 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895302 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884649 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961246 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112017 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797424 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947231 s, 11.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (23s) == sanityn test 77g: Change TBF type directly ============ 12:06:48 (1789142808) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (10s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 12:06:59 (1789142819) oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 12:07:10 (1789142830) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 12:07:33 (1789142853) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.593225 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100496 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105516 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108182 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162723 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226607 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146408 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239441 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245348 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222329 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280466 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314044 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303066 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283612 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290363 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284583 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294422 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230769 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225879 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233687 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238731 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244357 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243107 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249526 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295419 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271543 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297381 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283897 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288716 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312996 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295139 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303746 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363775 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.70028 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.584 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (66s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 12:08:39 (1789142919) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2811 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102027 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910217 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158697 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130822 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17364 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158158 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173729 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981792 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176368 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229872 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889865 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213183 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21752 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331907 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27921 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351825 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832779 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909049 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906117 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936641 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987192 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102637 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108077 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104401 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104884 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104639 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110825 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122811 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125478 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134796 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134675 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147746 s, 7.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9398 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9351 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28798 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100799 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757328 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109721 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130113 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171608 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588751 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219957 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133851 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147722 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153981 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275368 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165064 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276236 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230424 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306076 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278931 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215709 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258509 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630498 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641084 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738958 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385554 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427446 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373649 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848057 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494149 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436091 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462875 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053739 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478672 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517531 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573541 s, 18.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9484 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9698 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (150s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 12:11:09 (1789143069) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28144 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882773 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941741 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802161 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879185 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15218 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107991 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106828 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156517 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15148 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223514 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240497 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26881 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222151 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281332 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263322 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337665 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737773 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619875 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840754 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856912 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868331 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898784 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898773 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715869 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802423 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877691 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761809 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850953 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736902 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883057 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103664 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959479 s, 10.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9924 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9797 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32776 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941696 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128066 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903627 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110461 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13492 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253578 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241235 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245131 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220529 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284103 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307239 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296435 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315148 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309114 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282609 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366138 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887086 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988261 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856152 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876224 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109481 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11614 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109512 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111213 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123495 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963607 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1386 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902636 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110695 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881776 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119607 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108094 s, 9.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8833 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.954 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (152s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 12:13:41 (1789143221) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.462215 s, 36.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625758 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930329 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995498 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162963 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15986 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16654 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202868 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22681 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218736 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239872 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259626 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224906 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294353 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272482 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264832 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343367 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655454 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539585 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460184 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852504 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852306 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045958 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826226 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639306 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658368 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670245 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835873 s, 12.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642564 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091578 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840359 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808929 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105805 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.88683 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87374 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498344 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704098 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813988 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119559 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137538 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144277 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228449 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206228 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230996 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259893 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261178 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251731 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228333 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233405 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284052 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283921 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306024 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710996 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400644 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725719 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489074 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531509 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996083 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921982 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912396 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726743 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715956 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773315 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866791 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820792 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10155 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10221 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114823 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.97178 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91578 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441669 s, 38.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377576 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805975 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784322 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154098 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12774 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192605 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181437 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20735 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238142 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293192 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266147 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280003 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308072 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35687 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334869 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404182 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741238 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811331 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801231 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968179 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404517 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434597 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791341 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103549 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553684 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06228 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689654 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144687 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152774 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881332 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148548 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111072 s, 9.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.77617 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.35113 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.771941 s, 21.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898039 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745703 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104432 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131888 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130443 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149653 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183838 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172892 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874527 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842334 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264093 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248653 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208844 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267536 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384442 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279506 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795445 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373669 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486559 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116746 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502864 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797567 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568227 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681792 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131732 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938156 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808631 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103967 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841884 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873892 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091728 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924383 s, 11.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1034 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0476 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.52093 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91316 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.99295 s, 16.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811944 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093103 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095299 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113208 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828202 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113944 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192171 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186416 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209452 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207208 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172054 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16826 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235622 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141967 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173642 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212047 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050119 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547276 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580376 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463384 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499441 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530636 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385434 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516903 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592512 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669928 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790156 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597055 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068987 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710531 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797577 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080906 s, 13.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.787 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6131 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741697 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509704 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887869 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707469 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147464 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239789 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173024 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163107 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204247 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18768 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19015 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255173 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229432 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275208 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200959 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206756 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237251 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548107 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569446 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631893 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603422 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466039 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911354 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502559 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047898 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101084 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095339 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629519 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855412 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676898 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721388 s, 14.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108777 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797465 s, 13.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95881 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0556 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.656917 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316235 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423829 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613405 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130861 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173338 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175198 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150318 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181786 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162445 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168552 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191618 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18754 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189964 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200206 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192811 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190208 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085568 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691315 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10604 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448305 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806149 s, 13.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0055155 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409415 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467306 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564653 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345743 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397091 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318857 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531879 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305445 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654262 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586691 s, 17.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.613 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90366 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3805 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96037 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (435s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 12:20:56 (1789143656) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.573784 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121176 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929059 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104357 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146463 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174749 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339925 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307885 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400764 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418206 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41803 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376682 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423194 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360149 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345887 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429992 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500689 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11198 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123126 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11376 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427532 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110753 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114971 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559992 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870614 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155991 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892575 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773979 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145609 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160446 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102199 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991588 s, 10.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.12897 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94142 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (53s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 12:21:50 (1789143710) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633303 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144508 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156335 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172876 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413861 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412886 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353808 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385792 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368834 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454814 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481127 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489099 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524482 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507115 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394273 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4438 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.556578 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffffa091d940c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385246 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477372 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143696 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714695 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049732 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647898 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524258 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480097 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700487 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755839 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707737 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190833 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873065 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904109 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990643 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910363 s, 11.5 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.685 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.178 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 12:22:46 (1789143766) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467935 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702718 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080857 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106509 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178752 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172011 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156622 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18808 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219958 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215689 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266175 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284299 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266519 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26272 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268823 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339792 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333123 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091d940c800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351009 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464412 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314227 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196871 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230194 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544934 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288115 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731947 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579087 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040723 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648479 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406243 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458823 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467674 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526742 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556926 s, 18.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.89879 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3656 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 12:23:34 (1789143814) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.626454 s, 26.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100343 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623772 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110874 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10555 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876981 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137487 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192574 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15968 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202749 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145815 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203111 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149762 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276858 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269322 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273495 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345608 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522467 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275707 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440208 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551328 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476126 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081488 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882912 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924613 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853019 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859776 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107385 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120874 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115591 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124886 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127062 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137788 s, 7.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8007 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89519 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.58405 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96666 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 12:25:10 (1789143910) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03872 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620219 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119764 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184047 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179245 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2008 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23316 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247866 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308545 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292964 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335048 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327088 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3295 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338599 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396265 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401374 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451634 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142004 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122974 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945366 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107639 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884098 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12065 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917062 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095663 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100328 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283027 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840368 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303936 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902955 s, 11.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517098 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111229 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071995 s, 14.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9972 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8266 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (73s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 12:26:24 (1789143984) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618146 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781081 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886264 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102371 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169049 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167207 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228126 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116603 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214994 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146722 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148846 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172457 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173206 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154273 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179061 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206923 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236798 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618509 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765101 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673132 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788575 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128483 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950266 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071444 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943871 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098072 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111705 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101437 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115735 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128493 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117965 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132287 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161592 s, 6.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76312 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89636 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.56846 s, 29.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695362 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734113 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149596 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233179 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250574 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260529 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292686 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294055 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295502 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332968 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331319 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341709 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363467 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366266 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373226 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39691 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256712 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255079 s, 411 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104985 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527565 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746167 s, 141 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541413 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142806 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705436 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539501 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419493 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748166 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122977 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788012 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181827 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014612 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219777 s, 47.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5053 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94534 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7077 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87005 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (138s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 12:28:42 (1789144122) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:) Stopping client oleg432-client.virtnet /mnt/lustre opts: Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg432-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg432-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg432-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg432-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70066/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg432-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa091c39ad000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa091f4ee1000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091c39ad000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091f4ee1000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.775819 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160119 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224243 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861222 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266383 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295209 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297904 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224074 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293822 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379379 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415542 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458994 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393514 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504684 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.602687 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453274 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091f4ee1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa091f4ee1000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460358 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220256 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281101 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232336 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179897 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225001 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229706 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298684 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221552 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240716 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263123 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267133 s, 3.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258627 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263216 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27914 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284268 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291175 s, 3.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.48055 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24988 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.510299 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549341 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690297 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153551 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13727 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107569 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152059 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25616 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284668 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274258 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241372 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281422 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227366 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293857 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291543 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245742 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303134 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00937831 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155279 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421399 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119957 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594731 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00653896 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439863 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608593 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144475 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161936 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106576 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146531 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132519 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113956 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013645 s, 76.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366784 s, 28.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.532 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95803 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (210s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 12:32:12 (1789144332) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (21s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 12:32:33 (1789144353) On MGS 192.168.204.132, active = nodemap.active=1 On MGS 192.168.204.132, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567711 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403483 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797408 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702841 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161784 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21413 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110699 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449144 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196666 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856507 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104117 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837659 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319158 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35994 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360103 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374619 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440575 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091f4ee1000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821596 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885948 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996314 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129011 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138024 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102739 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123526 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116932 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119419 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136243 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154855 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120307 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131231 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150264 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140537 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14811 s, 7.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.55975 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0883 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.132, TBF.id = On MGS 192.168.204.132, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.616915 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908286 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488518 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844797 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751522 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112182 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830802 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332149 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1204 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182257 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201254 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110356 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173478 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102383 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213107 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091f4ee1000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa091f4ee1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069968 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339218 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250002 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749062 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281728 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568118 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387892 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824586 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268609 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013002 s, 80.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125114 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563598 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609421 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258721 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994289 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00854316 s, 123 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179388 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279738 s, 37.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.47309 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.37243 s, 44.2 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 On MGS 192.168.204.132, active = nodemap.active=1 On MGS 192.168.204.132, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.63351 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618995 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804162 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115594 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183252 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17678 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114668 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180056 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217615 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824081 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240426 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245058 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401631 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277043 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321488 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330537 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373207 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648453 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873132 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118281 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994537 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405968 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344657 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799979 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466571 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553093 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562687 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701905 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658911 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715342 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602262 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166971 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185817 s, 56.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.0531 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0666 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.132, TBF.id = On MGS 192.168.204.132, active = nodemap.active=0 PASS 77kj (182s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 12:35:35 (1789144535) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 12:35:45 (1789144545) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg432-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47842 s, 301 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg432-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1462 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 12:36:44 (1789144604) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.476991 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072624 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622902 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144142 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183683 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182384 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251889 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168757 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232265 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291482 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153447 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190862 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165567 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258111 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323352 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa091f4ee1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253073 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469774 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763296 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141477 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089557 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083758 s, 12.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902503 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121595 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720804 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159925 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148472 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110453 s, 9.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128287 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156577 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117263 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166826 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153247 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168858 s, 6.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93376 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.55178 s, 18.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.558766 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170752 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192939 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279386 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387348 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326031 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220139 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39504 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312054 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300424 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4312 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367687 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360789 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462098 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348326 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376905 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395734 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa091f4ee1000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffffa091f4ee1000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236307 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26777 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245287 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227977 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245274 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275209 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320562 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313945 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235332 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232507 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274964 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351957 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298712 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260351 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336442 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259601 s, 4.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.5875 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1992 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (97s) == sanityn test 77o: Changing rank should not panic ====== 12:38:22 (1789144702) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (16s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 12:38:39 (1789144719) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (122s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 12:40:41 (1789144841) mds.MDS.mdt.nrs_policies=tbf oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg432-client: oleg432-server: ssh exited with exit code 36 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 12:41:18 (1789144878) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273299 kill md_thread xargs PASS 77r (60s) == sanityn test 77s: Check TBF LRU shrinker ============== 12:42:18 (1789144938) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (68s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 12:43:26 (1789145006) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 12:43:37 (1789145017) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 12:43:39.000000000 -0400 Modify: 2026-09-11 12:43:39.000000000 -0400 Change: 2026-09-11 12:43:39.000000000 -0400 Birth: 2026-09-11 12:43:40.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 12:43:56 (1789145036) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 12:43:59 (1789145039) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 12:44:04 (1789145044) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 12:44:07 (1789145047) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 12:44:10 (1789145050) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 12:44:14 (1789145054) - open/close 773 (time 1789145070.64 total 10.01 last 77.22) - open/close 1408 (time 1789145080.65 total 20.02 last 63.44) - open/close 2006 (time 1789145090.69 total 30.07 last 59.53) total: 2500 open/close in 38.69 seconds: 64.61 ops/second snapshot_time 1789145195.889485199 secs.nsecs start_time 1789145058.750238093 secs.nsecs elapsed_time 137.139247106 secs.nsecs open 2505 samples [usecs] 169 28603 4178689 13827198971 close 2504 samples [usecs] 190 11210 941908 829847394 mknod 2501 samples [usecs] 547 28492 3642425 10968671963 unlink 2500 samples [usecs] 772 90911 6610504 84723165532 mkdir 2 samples [usecs] 4413 8555 12968 92662594 rmdir 2 samples [usecs] 14585 46512 61097 2376088369 rename 2500 samples [usecs] 541 99001 20249872 376536692010 getattr 3761 samples [usecs] 7 8441 250412 519540792 parallel_rename_file 2500 samples [usecs] 541 99001 20249872 376536692010 crossdir_rename 2500 samples [usecs] 541 99001 20249872 376536692010 readpage 34 samples [usecs] 5106 53598 903921 29003118823 PASS 81d (147s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 12:46:41 (1789145201) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7634 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7634 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 12:46:47 (1789145207) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 12:46:51 (1789145211) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 12:47:04 (1789145224) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789145226 Finishing test test1 at 1789145230 PASS 85 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 12:47:14 (1789145234) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 12:47:18 (1789145238) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 12:47:22 (1789145242) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 12:47:25 (1789145245) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5755 0x167b 0x240000400 1 5717 0x1655 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5756 0x167c 0x240000400 1 5718 0x1656 0x280000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 12:47:44 (1789145264) osc.lustre-OST0000-osc-ffffa091c39ad000.idle_timeout=0 osc.lustre-OST0000-osc-ffffa091f4ee1000.idle_timeout=0 osc.lustre-OST0001-osc-ffffa091c39ad000.idle_timeout=0 osc.lustre-OST0001-osc-ffffa091f4ee1000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.239347 s, 34.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99484 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290257 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0700723 s, 58.5 kB/s osc.lustre-OST0000-osc-ffffa091c39ad000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa091f4ee1000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091c39ad000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa091f4ee1000.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 12:48:01 (1789145281) llite.lustre-ffffa091c39ad000.fast_read=0 llite.lustre-ffffa091f4ee1000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0649833 s, 32.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffffa091c39ad000.fast_read=1 llite.lustre-ffffa091f4ee1000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 12:48:21 (1789145301) llite.lustre-ffffa091c39ad000.fast_read=0 llite.lustre-ffffa091f4ee1000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00366791 s, 3.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffffa091c39ad000.fast_read=1 llite.lustre-ffffa091f4ee1000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 12:48:35 (1789145315) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 12:48:39 (1789145319) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110949 s, 369 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 12:48:47 (1789145327) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15029 s, 14.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 12:48:55 (1789145335) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152041 s, 13.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 12:49:03 (1789145343) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00724973 s, 1.7 MB/s PASS 100e (10s) == sanityn test 101a: Discard DoM data on unlink ========= 12:49:13 (1789145353) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00833779 s, 491 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 12:49:21 (1789145361) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00738068 s, 555 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 12:49:29 (1789145369) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0060066 s, 682 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 12:49:38 (1789145378) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 12:49:45 (1789145385) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789145388 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789145392 Finishing test test23 at 1789145392 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 12:49:59 (1789145399) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 12:50:02 (1789145402) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 12:50:26 (1789145426) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 12:50:30 (1789145430) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00920165 s, 111 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 12:50:31.000000000 -0400 Modify: 2026-09-11 12:50:31.000000000 -0400 Change: 2026-09-11 12:50:31.000000000 -0400 Birth: 2026-09-11 12:50:32.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938183 -rw-r--r-- 1 0 0 1789145432 1789145431 1789145431 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 12:50:31.000000000 -0400 Modify: 2026-09-11 12:50:31.000000000 -0400 Change: 2026-09-11 12:50:31.000000000 -0400 Birth: 2026-09-11 12:50:32.000000000 -0400 1789145431 1024 1 PASS 106b (11s) == sanityn test 106c: Verify statx attributes mask ======= 12:50:40 (1789145440) PASS 106c (10s) == sanityn test 107a: Basic grouplock conflict =========== 12:50:50 (1789145450) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.453127 s, 23.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5765 0x1685 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 12:51:00 (1789145460) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351402 s, 29.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5725 0x165d 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7634 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 12:51:15 (1789145475) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00467656 s, 2.1 MB/s llite.lustre-ffffa091c39ad000.hybrid_io=0 llite.lustre-ffffa091f4ee1000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.74121 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 12:51:28 (1789145488) 192.168.204.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 192.168.204.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 PASS 109 (56s) == sanityn test 110: do not grant another lock on resend ========================================================== 12:52:25 (1789145545) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 12:52:28.000000000 -0400 Modify: 2026-09-11 12:52:28.000000000 -0400 Change: 2026-09-11 12:52:28.000000000 -0400 Birth: 2026-09-11 12:52:29.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg432-client.virtnet oleg432-client.virtnet PASS 110 (85s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 12:53:50 (1789145630) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 12:53:55 (1789145635) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 12:53:58 (1789145638) lustre-OST0000-osc-ffffa091c3ce7000 active. lustre-OST0001-osc-ffffa091c3ce7000 active. lustre-OST0000-osc-ffffa091c3ce7000 active. lustre-OST0001-osc-ffffa091c3ce7000 active. MGC192.168.204.132@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 12:54:07 (1789145647) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 12:54:11 (1789145651) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 12:54:14 (1789145654) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 12:54:18 (1789145658) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0333124 s, 615 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.110365 s, 186 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 12:54:29 (1789145669) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=10 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.175 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1221 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.314 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1862 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2547 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2003 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2213 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2508 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0495 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.225 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2663 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3228 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2946 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3313 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3865 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.325 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1767 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3673 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4266 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3991 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa091c3ce7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa091c3ce7000.ost_server_uuid in FULL state after 0 sec PASS 200 (84s) cleanup: ====================================================== == sanityn test complete, duration 10030 sec ============= 12:55:54 (1789145754) osc.lustre-OST0000-osc-ffffa091c3ce7000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0000-osc-ffffa091e00e1000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffffa091c3ce7000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffffa091e00e1000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789145759.200336828 secs.nsecs start_time 1789144186.017848101 secs.nsecs elapsed_time 1573.182488727 secs.nsecs uncompressed_objects 1451 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789145759.200399329 secs.nsecs start_time 1789144201.705925491 secs.nsecs elapsed_time 1557.494473838 secs.nsecs uncompressed_objects 1498 samples [reqs] === sanityn: start cleanup 12:55:58 (1789145758) === Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 12:59:56 (1789145996) ===