-----============= acceptance-small: sanityn ============----- Tue Sep 15 14:34:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg329-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 14:34:39 (1789497279) === Starting client oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Mount client oleg329-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Started clients oleg329-client.virtnet: 192.168.203.129@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg329-client.virtnet: executing check_config_client /mnt/lustre oleg329-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg329-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9db3c32fd000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9db3ca0cb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3c32fd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3ca0cb000.idle_timeout=debug disable quota as required oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:35:11 (1789497311) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238513 s, 44.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:35:14 (1789497314) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:35:21 (1789497321) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:35:27 (1789497327) -rw-r--r-- 1 root root 0 Sep 15 14:35 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:35:34 (1789497334) -rw-r--r-- 1 root root 0 Sep 15 14:35 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:35:40 (1789497340) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:35:46 (1789497346) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:35:52 (1789497352) total 1 -rwxrwxrwx 1 root root 0 Sep 15 14:35 f2a -rwxrwxrwx 1 root root 0 Sep 15 14:35 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 15 14:35 f2a -rwxrwxrwx 1 root root 0 Sep 15 14:35 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0138359 s, 37.0 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:35:59 (1789497359) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 14:36:02 (1789497362) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.373588 s, 5.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:36:08 (1789497368) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:36:14 (1789497374) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:36:21 (1789497381) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 14:36:27 (1789497387) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:36:33 (1789497393) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:36:38 (1789497398) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:36:44 (1789497404) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 14:36:50 (1789497410) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603617 s, 17.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0458528 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:36:55 (1789497415) 1+0 records in 1+0 records out 1 byte copied, 0.00791369 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172754 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00816452 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00931745 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00977973 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112053 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113368 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172457 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00727941 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00566242 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100171 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00788825 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:37:01 (1789497421) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.011172 s, 275 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0494928 s, 82.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000510452 s, 6.0 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:37:06 (1789497426) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:37:12 (1789497432) warning: '-runtime' deprecated, use '-t runtime' instead - link 356 (time 1789497444.40 total 10.02 last 35.55) - link 697 (time 1789497454.41 total 20.03 last 34.06) - link 1032 (time 1789497464.41 total 30.03 last 33.50) - link 1370 (time 1789497474.41 total 40.03 last 33.79) - link 1644 (time 1789497484.46 total 50.08 last 27.26) - link 2021 (time 1789497494.48 total 60.10 last 37.63) - link 2354 (time 1789497504.50 total 70.11 last 33.25) - link 2674 (time 1789497514.53 total 80.15 last 31.90) - link 3018 (time 1789497524.54 total 90.16 last 34.36) total: 3338 link in 100.00 seconds: 33.38 ops/second using seed 3460255261 running for 100 seconds - stat 5919 (time 1789497446 ; total 11 ; last 11) - stat 10000 (time 1789497454 ; total 19 ; last 8) - stat 15737 (time 1789497465 ; total 30 ; last 11) - stat 20000 (time 1789497473 ; total 38 ; last 8) - stat 25002 (time 1789497484 ; total 49 ; last 11) - stat 28726 (time 1789497495 ; total 60 ; last 11) - stat 30000 (time 1789497499 ; total 64 ; last 4) - stat 32652 (time 1789497510 ; total 75 ; last 11) - stat 35788 (time 1789497521 ; total 86 ; last 11) - stat 38443 (time 1789497532 ; total 97 ; last 11) total: 39431 stats in 100 seconds: 394.309998 stats/second - unlinked 0 (time 1789497485 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3338) error: No such file or directory total: 3337 unlinks in 86 seconds: 38.802326 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21427) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21429) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 14:39:36 (1789497576) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:39:45 (1789497585) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22613 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:39:52 (1789497592) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23192 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:39:58 (1789497598) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23770 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:40:04 (1789497604) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24352 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:40:10 (1789497610) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24934 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:40:17 (1789497617) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (49s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:41:06 (1789497666) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.624871 s, 13.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.389106 s, 21.6 MB/s Chance of close/open is 1 in 50 Seed set to 7670 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[0] 1789497671.617980 mapwrite 0x45c4a5 thru 0x46b4b3 (0xf00f bytes) 000005[1] 1789497671.778509 read 0x16534e thru 0x16a000 (0x4cb3 bytes) 000008[0] 1789497671.831481 mapwrite 0x03d254 thru 0x04b353 (0xe100 bytes) 000009[1] 1789497671.971038 punch from 0x0dd195 to 0x0df07b (0x1ee7 bytes) truncating to largest ever: 0x27155d 000010[0] 1789497672.009883 trunc from 0x46b4b4 to 0x27155c (0x1f9f57 bytes) 000011[1] 1789497672.048468 punch from 0x02d35f to 0x03a602 (0xd2a4 bytes) 000014[1] 1789497672.113438 mapwrite 0x37d3e8 thru 0x386f22 (0x9b3b bytes) 000015[0] 1789497672.233056 mapwrite 0x7b1b5d thru 0x7c087a (0xed1e bytes) 000016[1] 1789497672.512121 mapread 0x678aab thru 0x67b7f7 (0x2d4d bytes) 000017[0] 1789497672.565480 mapread 0x51481d thru 0x519199 (0x497d bytes) 000018[0] 1789497672.588921 write 0x0d5590 thru 0x0da59c (0x500d bytes) 000019[1] 1789497672.612306 write 0x7a5db0 thru 0x7ad8b2 (0x7b03 bytes) 000021[0] 1789497672.658567 mapread 0x36039d thru 0x36af33 (0xab97 bytes) 000023[1] 1789497672.709949 read 0x28ccac thru 0x28cee5 (0x23a bytes) 000025[0] 1789497672.759239 trunc from 0x7c087b to 0x10cf76 (0x6b3904 bytes) 000026[0] 1789497672.831511 write 0x22685c thru 0x226faf (0x754 bytes) 000027[0] 1789497672.867016 read 0x17326e thru 0x17522f (0x1fc2 bytes) 000031[1] 1789497672.920596 read 0x084857 thru 0x089d83 (0x552d bytes) 000032[1] 1789497673.020085 mapwrite 0x7e9052 thru 0x7ee964 (0x5913 bytes) 000033[0] 1789497673.228708 write 0x80de3e thru 0x819dfa (0xbfbd bytes) 000034[0] 1789497673.260520 punch from 0x5cf22a to 0x5d42e9 (0x50c0 bytes) 000036[1] 1789497673.300418 write 0x5348db thru 0x53931e (0x4a44 bytes) 000037[1] 1789497673.338986 punch from 0x019136 to 0x01fbc0 (0x6a8b bytes) 000039[0] 1789497673.410872 write 0x65ff63 thru 0x662846 (0x28e4 bytes) 000040[1] 1789497673.496834 write 0x84bd78 thru 0x84d048 (0x12d1 bytes) 000041[1] 1789497673.523724 punch from 0x7b7ea5 to 0x7c45ca (0xc726 bytes) 000042[1] 1789497673.558793 trunc from 0x84d049 to 0x0f8da0 (0x7542a8 bytes) 000044[0] 1789497673.623624 punch from 0x0d29c6 to 0x0d7647 (0x4c82 bytes) 000045[0] 1789497673.654668 trunc from 0x0f8da1 to 0x108faa (0x1020a bytes) 000046[1] 1789497673.688340 read 0x085804 thru 0x08fc17 (0xa414 bytes) 000049[1] 1789497673.794484 read 0x08cdd1 thru 0x090a9e (0x3cce bytes) 000050[0] 1789497673.799609 read 0x047587 thru 0x04cfb8 (0x5a32 bytes) 000052[1] 1789497673.826503 write 0x761a98 thru 0x76947d (0x79e6 bytes) 000053[1] 1789497673.863937 mapwrite 0x84f197 thru 0x853339 (0x41a3 bytes) 000054[1] 1789497674.098089 mapwrite 0x5a00fa thru 0x5ad5bf (0xd4c6 bytes) truncating to largest ever: 0x6cd430 000055[0] 1789497674.207912 trunc from 0x85333a to 0x6cd42f (0x185f0a bytes) 000056[0] 1789497674.299862 trunc from 0x6cd430 to 0x1baca7 (0x512788 bytes) 000057[0] 1789497674.327509 mapwrite 0x55e9ed thru 0x56e891 (0xfea5 bytes) 000060[1] 1789497674.527002 mapwrite 0x5f41f8 thru 0x5fe4f8 (0xa301 bytes) truncating to largest ever: 0x8fcee7 000062[1] 1789497674.728592 trunc from 0x5fe4f9 to 0x8fcee6 (0x2fe9ee bytes) 000063[1] 1789497674.756161 mapwrite 0x381a20 thru 0x389813 (0x7df4 bytes) 000065[1] 1789497674.878790 mapread 0x4f31a4 thru 0x4f8293 (0x50f0 bytes) 000066[1] 1789497674.922494 mapread 0x15d4bf thru 0x16c48f (0xefd1 bytes) 000067[0] 1789497675.005630 mapwrite 0x98078f thru 0x982b2a (0x239c bytes) 000068[1] 1789497675.195850 punch from 0x96ee7a to 0x97dfda (0xf161 bytes) 000072[0] 1789497675.279586 trunc from 0x982b2b to 0x34a655 (0x6384d5 bytes) 000073[1] 1789497675.332081 mapwrite 0x368628 thru 0x36f3b3 (0x6d8c bytes) 000074[1] 1789497675.462068 punch from 0x252e12 to 0x25ed66 (0xbf55 bytes) 000075[1] 1789497675.473436 mapwrite 0x265fea thru 0x268926 (0x293d bytes) 000076[0] 1789497675.583268 mapwrite 0x806a9a thru 0x807d7c (0x12e3 bytes) 000077[0] 1789497675.727171 write 0x5d9edc thru 0x5e4a5e (0xab83 bytes) 000078[1] 1789497675.746264 mapread 0x1e60a1 thru 0x1f44fe (0xe45e bytes) 000079[0] 1789497675.889554 mapwrite 0x471ece thru 0x477264 (0x5397 bytes) 000080[1] 1789497676.059985 punch from 0x411f0b to 0x414021 (0x2117 bytes) 000081[0] 1789497676.102628 mapread 0x4946ab thru 0x49fe1c (0xb772 bytes) 000083[0] 1789497676.244643 write 0x508085 thru 0x5116c1 (0x963d bytes) 000084[1] 1789497676.279231 mapwrite 0x1346ba thru 0x140d42 (0xc689 bytes) 000085[0] 1789497676.540554 trunc from 0x807d7d to 0x8876cf (0x7f953 bytes) 000086[0] 1789497676.605195 mapwrite 0x8c581d thru 0x8ceef8 (0x96dc bytes) 000087[0] 1789497676.758711 punch from 0x3cf0b8 to 0x3d5cb1 (0x6bfa bytes) 000090[1] 1789497676.809523 mapread 0x0c8d07 thru 0x0d320d (0xa507 bytes) 000091[0] 1789497676.919185 read 0x0d1b52 thru 0x0e00c4 (0xe573 bytes) 000092[0] 1789497676.937293 mapwrite 0x1b2341 thru 0x1b8afe (0x67be bytes) 000095[1] 1789497677.067044 read 0x3e59a2 thru 0x3f1cc5 (0xc324 bytes) 000097[1] 1789497677.147827 write 0x4def2d thru 0x4e1372 (0x2446 bytes) 000100[0] 1789497677.279305 trunc from 0x8ceef9 to 0x5d1b99 (0x2fd35f bytes) 000101[1] 1789497677.373284 read 0x4f1b55 thru 0x4f6d3e (0x51ea bytes) 000102[0] 1789497677.410561 trunc from 0x5d1b9a to 0x5466bb (0x8b4de bytes) 000104[0] 1789497677.455538 punch from 0x503389 to 0x5054d5 (0x214d bytes) 000105[0] 1789497677.467099 read 0x26cd5b thru 0x27b16e (0xe414 bytes) 000106[0] 1789497677.482930 punch from 0x28b421 to 0x2998f9 (0xe4d9 bytes) 000107[1] 1789497677.557326 mapwrite 0x2c0c72 thru 0x2cf6af (0xea3e bytes) 000109[1] 1789497677.774894 write 0x9a4334 thru 0x9acdc1 (0x8a8e bytes) 000112[1] 1789497677.824531 write 0x06caad thru 0x07540c (0x8960 bytes) 000115[1] 1789497677.899832 mapwrite 0x7e7e9a thru 0x7e8de7 (0xf4e bytes) 000116[1] 1789497678.095998 trunc from 0x9acdc2 to 0x12b7d8 (0x8815e9 bytes) 000117[1] 1789497678.218001 mapwrite 0x4ce6ff thru 0x4d0f81 (0x2883 bytes) 000118[0] 1789497678.434256 mapread 0x11661d thru 0x1166de (0xc2 bytes) 000119[0] 1789497678.525818 mapwrite 0x54ed6f thru 0x54efbd (0x24f bytes) 000120[1] 1789497678.694488 mapread 0x368fa7 thru 0x377b48 (0xeba2 bytes) 000121[0] 1789497678.825636 punch from 0x0f9b51 to 0x108e30 (0xf2e0 bytes) 000122[0] 1789497678.848047 trunc from 0x54efbe to 0x624764 (0xd57a7 bytes) 000123[0] 1789497678.888207 trunc from 0x624765 to 0x093dfd (0x590967 bytes) 000124[0] 1789497678.953882 trunc from 0x093dfe to 0x7e2ef2 (0x74f0f5 bytes) 000126[0] 1789497679.015424 write 0x64c185 thru 0x6591e5 (0xd061 bytes) 000127[1] 1789497679.027546 punch from 0x47337d to 0x47df74 (0xabf8 bytes) 000128[0] 1789497679.060676 trunc from 0x7e2ef3 to 0x7cf3cc (0x13b26 bytes) 000129[1] 1789497679.091314 write 0x22838f thru 0x22f19e (0x6e10 bytes) 000130[0] 1789497679.150710 trunc from 0x7cf3cd to 0x3c4a46 (0x40a986 bytes) 000131[1] 1789497679.236593 trunc from 0x3c4a47 to 0x024160 (0x3a08e6 bytes) 000133[1] 1789497679.306570 mapread 0x012433 thru 0x0213f2 (0xefc0 bytes) 000134[0] 1789497679.335495 mapwrite 0x275bfc thru 0x278e53 (0x3258 bytes) 000135[0] 1789497679.475460 mapread 0x1da34f thru 0x1e4c8d (0xa93f bytes) 000137[1] 1789497679.570599 mapread 0x1ec021 thru 0x1f1b11 (0x5af1 bytes) 000138[0] 1789497679.673531 write 0x410ec5 thru 0x41cfe4 (0xc120 bytes) 000139[1] 1789497679.677572 read 0x23e670 thru 0x2443eb (0x5d7c bytes) 000140[1] 1789497679.798230 mapwrite 0x19310b thru 0x19a6df (0x75d5 bytes) 000141[0] 1789497680.009611 punch from 0x182092 to 0x188f18 (0x6e87 bytes) 000142[1] 1789497680.053031 trunc from 0x41cfe5 to 0x54627b (0x129297 bytes) 000144[0] 1789497680.276035 mapread 0x2e992f thru 0x2ebdfa (0x24cc bytes) 000145[0] 1789497680.369501 punch from 0x14450c to 0x14933a (0x4e2f bytes) 000146[1] 1789497680.422502 read 0x330323 thru 0x33bbb9 (0xb897 bytes) 000153[1] 1789497680.545394 mapwrite 0x9f6462 thru 0x9f8829 (0x23c8 bytes) 000156[1] 1789497680.720580 trunc from 0x9f882a to 0x654eac (0x3a397d bytes) 000158[1] 1789497680.782046 punch from 0x03bba5 to 0x03c50d (0x969 bytes) 000163[0] 1789497680.837516 mapwrite 0x6691c3 thru 0x66b40d (0x224b bytes) 000165[1] 1789497681.133783 trunc from 0x66b40e to 0x4525d2 (0x218e3b bytes) 000166[1] 1789497681.219558 trunc from 0x4525d3 to 0x441f07 (0x106cb bytes) 000167[1] 1789497681.260064 punch from 0x2b0cc1 to 0x2b9d17 (0x9057 bytes) 000168[0] 1789497681.282917 trunc from 0x441f08 to 0x77198a (0x32fa83 bytes) 000174[0] 1789497681.407700 read 0x431788 thru 0x4379e9 (0x6262 bytes) 000175[1] 1789497681.449262 punch from 0x6c8005 to 0x6c8df0 (0xdec bytes) 000176[0] 1789497681.498411 mapread 0x03786d thru 0x040d42 (0x94d6 bytes) 000180[0] 1789497681.658805 write 0x0f98f0 thru 0x104c14 (0xb325 bytes) 000181[1] 1789497681.699217 read 0x0ec739 thru 0x0f4cbd (0x8585 bytes) 000184[0] 1789497681.746899 mapread 0x5bbcb1 thru 0x5c5d1f (0xa06f bytes) 000185[0] 1789497681.819981 write 0x85e56e thru 0x86248b (0x3f1e bytes) 000186[1] 1789497681.842551 read 0x529419 thru 0x538074 (0xec5c bytes) 000187[1] 1789497681.879196 read 0x68e8e8 thru 0x69abb2 (0xc2cb bytes) 000188[1] 1789497681.908289 mapwrite 0x144168 thru 0x14569e (0x1537 bytes) 000189[0] 1789497682.051540 read 0x2a0f5a thru 0x2a8582 (0x7629 bytes) 000190[0] 1789497682.099460 trunc from 0x86248c to 0x680663 (0x1e1e28 bytes) 000193[0] 1789497682.245279 trunc from 0x680664 to 0x3a4b1d (0x2dbb46 bytes) 000194[1] 1789497682.277112 trunc from 0x3a4b1e to 0x0fe3d3 (0x2a674a bytes) 000195[1] 1789497682.350530 read 0x0cdcc0 thru 0x0dce9c (0xf1dd bytes) 000196[1] 1789497682.408218 read 0x063bd0 thru 0x070e9a (0xd2cb bytes) 000197[1] 1789497682.437950 punch from 0x0a13f6 to 0x0a818a (0x6d95 bytes) 000198[1] 1789497682.470673 write 0x95bdaf thru 0x96b010 (0xf262 bytes) 000199[1] 1789497682.485487 punch from 0x71573c to 0x71f148 (0x9a0d bytes) 000202[0] 1789497682.511072 write 0x7ff3ae thru 0x807a7c (0x86cf bytes) 000203[1] 1789497682.580472 read 0x928da3 thru 0x930fda (0x8238 bytes) 000204[1] 1789497682.679509 mapwrite 0x55678e thru 0x5570b4 (0x927 bytes) 000205[1] 1789497682.835031 mapread 0x23d22a thru 0x23ef5a (0x1d31 bytes) 000206[0] 1789497682.848611 write 0x362e31 thru 0x36f701 (0xc8d1 bytes) 000207[0] 1789497682.897558 mapwrite 0x0f004a thru 0x0f9a8a (0x9a41 bytes) 000209[1] 1789497683.082387 mapwrite 0x67ccd3 thru 0x67fe44 (0x3172 bytes) 000210[1] 1789497683.262262 write 0x87bd67 thru 0x87c3bd (0x657 bytes) 000212[0] 1789497683.360384 write 0x37ea51 thru 0x38d617 (0xebc7 bytes) 000215[1] 1789497683.483661 trunc from 0x96b011 to 0x4d714f (0x493ec1 bytes) 000217[0] 1789497683.603493 write 0x2a30bc thru 0x2a4381 (0x12c6 bytes) 000219[0] 1789497683.656113 punch from 0x123fd4 to 0x133095 (0xf0c2 bytes) 000220[0] 1789497683.674389 write 0x6a5070 thru 0x6aa220 (0x51b1 bytes) 000222[1] 1789497683.692937 read 0x1d1414 thru 0x1d81d8 (0x6dc5 bytes) 000223[1] 1789497683.739943 trunc from 0x6aa221 to 0x3ee977 (0x2bb8a9 bytes) 000225[0] 1789497683.853907 read 0x2bedaa thru 0x2c0dbe (0x2015 bytes) 000226[0] 1789497683.947016 punch from 0x0bc98e to 0x0c613b (0x97ae bytes) 000227[1] 1789497683.978952 trunc from 0x3ee978 to 0x187777 (0x267200 bytes) 000228[1] 1789497684.027907 trunc from 0x187778 to 0x1b7ca1 (0x3052a bytes) 000229[1] 1789497684.059816 mapwrite 0x311eee thru 0x31cbf6 (0xad09 bytes) 000234[1] 1789497684.235975 punch from 0x1c03f7 to 0x1c31a4 (0x2dae bytes) 000235[1] 1789497684.240084 trunc from 0x31cbf7 to 0x7ca36f (0x4ad779 bytes) 000238[0] 1789497684.369667 trunc from 0x7ca370 to 0x5ea3f8 (0x1dff77 bytes) 000239[1] 1789497684.442806 trunc from 0x5ea3f9 to 0x7d8078 (0x1edc80 bytes) 000240[0] 1789497684.510418 read 0x626c1a thru 0x629ebe (0x32a5 bytes) 000244[1] 1789497684.611999 mapwrite 0x86e3dd thru 0x878589 (0xa1ad bytes) 000245[0] 1789497684.739949 mapread 0x71a745 thru 0x722e0f (0x86cb bytes) 000246[0] 1789497684.797348 read 0x318549 thru 0x32853e (0xfff6 bytes) 000248[0] 1789497684.828492 write 0x723901 thru 0x7240e6 (0x7e6 bytes) 000252[1] 1789497684.874228 mapwrite 0x8dd4f7 thru 0x8e43a9 (0x6eb3 bytes) 000254[1] 1789497685.018559 punch from 0x07de1f to 0x08cf04 (0xf0e6 bytes) 000256[1] 1789497685.024470 mapwrite 0x0a43fa thru 0x0a6408 (0x200f bytes) 000257[1] 1789497685.099117 read 0x0d84fb thru 0x0db874 (0x337a bytes) 000258[0] 1789497685.100157 punch from 0x36a0df to 0x375f33 (0xbe55 bytes) 000259[0] 1789497685.140540 trunc from 0x8e43aa to 0x87c211 (0x68198 bytes) 000261[1] 1789497685.240277 punch from 0x0345b8 to 0x03926e (0x4cb7 bytes) 000264[0] 1789497685.340517 mapwrite 0x53087a thru 0x5334af (0x2c36 bytes) 000265[1] 1789497685.419966 mapwrite 0x99b28f thru 0x9a5812 (0xa584 bytes) 000266[1] 1789497685.555071 write 0x9b6fb1 thru 0x9c5b8c (0xebdc bytes) 000267[0] 1789497685.561830 mapwrite 0x9bcceb thru 0x9c3b5a (0x6e70 bytes) 000269[1] 1789497685.674617 trunc from 0x9c5b8d to 0x64bc07 (0x379f85 bytes) 000270[1] 1789497685.702911 mapwrite 0x9bd8b0 thru 0x9c099e (0x30ef bytes) 000272[1] 1789497685.833376 mapread 0x661ac0 thru 0x66c122 (0xa663 bytes) 000276[0] 1789497685.931042 write 0x9e2196 thru 0x9eb91f (0x978a bytes) 000279[0] 1789497685.998047 trunc from 0x9eb920 to 0x8c511c (0x126803 bytes) 000280[0] 1789497686.046120 mapwrite 0x4ee18e thru 0x4fc129 (0xdf9c bytes) 000281[0] 1789497686.167192 trunc from 0x8c511d to 0x218d96 (0x6ac386 bytes) 000283[0] 1789497686.255396 trunc from 0x218d97 to 0x619e33 (0x40109d bytes) 000284[1] 1789497686.301431 mapread 0x2ba6f4 thru 0x2be801 (0x410e bytes) 000285[1] 1789497686.372437 trunc from 0x619e34 to 0x5e7adf (0x32354 bytes) 000288[0] 1789497686.514946 punch from 0x1416c6 to 0x14586f (0x41aa bytes) 000291[1] 1789497686.586861 mapwrite 0x1808d1 thru 0x1884e9 (0x7c19 bytes) 000292[0] 1789497686.684242 mapread 0x2be117 thru 0x2c41e4 (0x60ce bytes) 000293[1] 1789497686.812209 write 0x5cb50f thru 0x5d1c5c (0x674e bytes) 000294[0] 1789497686.842880 write 0x207f32 thru 0x20af1e (0x2fed bytes) 000297[0] 1789497686.972428 write 0x8cfe0d thru 0x8daae3 (0xacd7 bytes) 000300[0] 1789497687.027952 read 0x21dc6c thru 0x2224cd (0x4862 bytes) 000301[0] 1789497687.049754 read 0x4faf10 thru 0x5036c9 (0x87ba bytes) 000305[0] 1789497687.104706 punch from 0x67b624 to 0x6830a0 (0x7a7d bytes) 000306[0] 1789497687.145400 trunc from 0x8daae4 to 0x5919e1 (0x349102 bytes) 000307[0] 1789497687.204064 mapread 0x58bfa4 thru 0x5919e1 (0x5a3e bytes) 000308[1] 1789497687.229896 trunc from 0x5919e2 to 0x2bd017 (0x2d49ca bytes) 000309[1] 1789497687.311330 mapread 0x05a929 thru 0x068c65 (0xe33d bytes) 000310[1] 1789497687.367047 write 0x289404 thru 0x28d99f (0x459c bytes) 000311[0] 1789497687.382796 read 0x14290d thru 0x143877 (0xf6b bytes) 000314[1] 1789497687.496597 trunc from 0x2bd018 to 0x03439a (0x288c7d bytes) 000316[0] 1789497687.591837 trunc from 0x03439b to 0x168b85 (0x1347eb bytes) 000319[1] 1789497687.683598 trunc from 0x168b86 to 0x38981b (0x220c96 bytes) 000321[0] 1789497687.726530 read 0x00657b thru 0x00d313 (0x6d99 bytes) 000322[0] 1789497687.792580 mapread 0x3278f8 thru 0x32a8fd (0x3006 bytes) 000324[0] 1789497687.844638 mapwrite 0x686b91 thru 0x6958f1 (0xed61 bytes) 000325[0] 1789497688.025809 read 0x0b8eb3 thru 0x0b9259 (0x3a7 bytes) 000326[1] 1789497688.031486 punch from 0x1eda08 to 0x1ee44e (0xa47 bytes) 000327[1] 1789497688.060407 write 0x96cfe4 thru 0x971dce (0x4deb bytes) 000328[0] 1789497688.076087 trunc from 0x971dcf to 0x8bbbbb (0xb6213 bytes) truncating to largest ever: 0x9eef2e 000329[0] 1789497688.109834 trunc from 0x8bbbbc to 0x9eef2d (0x133372 bytes) 000331[1] 1789497688.148111 write 0x6f8d79 thru 0x7071c7 (0xe44f bytes) 000333[0] 1789497688.197863 read 0x66f707 thru 0x67cfc6 (0xd8c0 bytes) 000336[0] 1789497688.260353 read 0x5a36c9 thru 0x5af47c (0xbdb4 bytes) 000337[0] 1789497688.273098 punch from 0x08d681 to 0x092759 (0x50d9 bytes) 000338[1] 1789497688.289735 write 0x905b9e thru 0x90cd93 (0x71f6 bytes) 000340[0] 1789497688.319440 trunc from 0x9eef2e to 0x915b5d (0xd93d0 bytes) 000342[0] 1789497688.400901 mapwrite 0x942bc3 thru 0x952b51 (0xff8f bytes) 000343[0] 1789497688.522913 read 0x364ab9 thru 0x366e7c (0x23c4 bytes) 000344[1] 1789497688.528720 punch from 0x49553d to 0x49cc2f (0x76f3 bytes) 000345[0] 1789497688.564327 mapwrite 0x59e278 thru 0x5ad020 (0xeda9 bytes) 000347[0] 1789497688.688931 read 0x1063fb thru 0x113736 (0xd33c bytes) 000348[1] 1789497688.713014 write 0x795b28 thru 0x7a34ff (0xd9d8 bytes) 000349[0] 1789497688.752511 mapread 0x19b4b3 thru 0x1aab41 (0xf68f bytes) 000350[0] 1789497688.809469 mapread 0x1203cd thru 0x123031 (0x2c65 bytes) 000351[1] 1789497688.844787 punch from 0x0c198b to 0x0c85f7 (0x6c6d bytes) 000356[0] 1789497688.904234 write 0x719945 thru 0x71a551 (0xc0d bytes) 000360[0] 1789497688.990074 read 0x3df03f thru 0x3e6031 (0x6ff3 bytes) 000361[1] 1789497689.006837 mapread 0x1e44cf thru 0x1ee6f2 (0xa224 bytes) 000362[0] 1789497689.041378 trunc from 0x952b52 to 0x58bc9f (0x3c6eb2 bytes) 000363[0] 1789497689.077672 mapread 0x2f429d thru 0x3022ea (0xe04e bytes) 000364[1] 1789497689.141512 trunc from 0x58bca0 to 0x10b197 (0x480b08 bytes) 000365[0] 1789497689.206394 mapread 0x0b855f thru 0x0c76d4 (0xf176 bytes) 000366[0] 1789497689.264957 mapwrite 0x4c32d2 thru 0x4d2666 (0xf395 bytes) 000367[0] 1789497689.397832 mapread 0x19c9ea thru 0x1a9b0b (0xd122 bytes) 000369[1] 1789497689.455348 punch from 0x05d8c0 to 0x063585 (0x5cc6 bytes) 000370[0] 1789497689.497011 trunc from 0x4d2667 to 0x16e4f0 (0x364176 bytes) 000378[0] 1789497689.701424 mapwrite 0x89da76 thru 0x8a226a (0x47f5 bytes) 000379[0] 1789497689.887553 read 0x1f6ce3 thru 0x1f7044 (0x362 bytes) 000380[1] 1789497689.894506 punch from 0x799618 to 0x799ebf (0x8a8 bytes) 000381[1] 1789497689.927429 mapread 0x6f24c2 thru 0x6fa196 (0x7cd5 bytes) 000382[0] 1789497689.980688 read 0x27981c thru 0x27d4dd (0x3cc2 bytes) 000383[0] 1789497690.004757 punch from 0x243d2b to 0x247507 (0x37dd bytes) 000384[1] 1789497690.025718 write 0x03fa70 thru 0x04bc36 (0xc1c7 bytes) 000385[0] 1789497690.095436 punch from 0x1f74ac to 0x2041d8 (0xcd2d bytes) 000386[1] 1789497690.141261 read 0x7db7e9 thru 0x7ea98f (0xf1a7 bytes) 000388[0] 1789497690.185466 punch from 0x7f5130 to 0x803fc5 (0xee96 bytes) 000389[1] 1789497690.252245 mapread 0x472c31 thru 0x4767a5 (0x3b75 bytes) 000390[0] 1789497690.312489 punch from 0x25b59a to 0x267349 (0xbdb0 bytes) 000392[1] 1789497690.334697 mapread 0x638882 thru 0x643c97 (0xb416 bytes) 000394[1] 1789497690.370115 mapread 0x26ac9f thru 0x273d37 (0x9099 bytes) 000398[0] 1789497690.411993 punch from 0x706d44 to 0x714055 (0xd312 bytes) 000399[0] 1789497690.430155 read 0x69c2c7 thru 0x6a8be4 (0xc91e bytes) 000401[1] 1789497690.458486 punch from 0x30222d to 0x311caa (0xfa7e bytes) 000403[1] 1789497690.519992 write 0x67d6a2 thru 0x68a9d2 (0xd331 bytes) 000404[0] 1789497690.568399 mapread 0x06b8b9 thru 0x07903e (0xd786 bytes) 000406[1] 1789497690.697530 punch from 0x2cb5b3 to 0x2ce67b (0x30c9 bytes) 000408[1] 1789497690.775227 punch from 0x3a2dee to 0x3b0a03 (0xdc16 bytes) 000411[1] 1789497690.818351 mapread 0x1c660c thru 0x1c857c (0x1f71 bytes) 000412[1] 1789497690.864634 mapread 0x37cef4 thru 0x38bbcd (0xecda bytes) 000415[1] 1789497690.948472 trunc from 0x8a226b to 0x4068f9 (0x49b971 bytes) 000417[0] 1789497690.996697 trunc from 0x4068fa to 0x62affb (0x224702 bytes) 000418[1] 1789497691.036888 read 0x0d2e04 thru 0x0d3a58 (0xc55 bytes) 000419[0] 1789497691.057978 read 0x39f26e thru 0x3a04f3 (0x1286 bytes) 000421[0] 1789497691.063433 trunc from 0x62affc to 0x776e85 (0x14be8a bytes) 000424[0] 1789497691.155580 write 0x114212 thru 0x11ef67 (0xad56 bytes) 000425[1] 1789497691.168440 read 0x099dbe thru 0x0a59b8 (0xbbfb bytes) 000426[0] 1789497691.202718 trunc from 0x776e86 to 0x8006ac (0x89827 bytes) 000427[1] 1789497691.263938 mapread 0x12e8aa thru 0x12f0e7 (0x83e bytes) 000429[0] 1789497691.326802 trunc from 0x8006ad to 0x985635 (0x184f89 bytes) 000430[0] 1789497691.372456 write 0x3f49cf thru 0x404054 (0xf686 bytes) 000431[1] 1789497691.386746 mapread 0x399131 thru 0x3a16c9 (0x8599 bytes) 000432[0] 1789497691.458466 write 0x35f983 thru 0x36cc4a (0xd2c8 bytes) 000434[0] 1789497691.519115 mapread 0x0e3532 thru 0x0f00ad (0xcb7c bytes) 000435[1] 1789497691.613371 read 0x6bcb09 thru 0x6c6f2d (0xa425 bytes) 000436[0] 1789497691.664980 mapread 0x38de19 thru 0x3984c1 (0xa6a9 bytes) 000437[1] 1789497691.720096 write 0x7a69f1 thru 0x7aece8 (0x82f8 bytes) 000439[0] 1789497691.815331 write 0x140abf thru 0x14b0b7 (0xa5f9 bytes) 000443[0] 1789497691.877957 read 0x61b3ac thru 0x6273ca (0xc01f bytes) 000445[1] 1789497691.934984 punch from 0x258f83 to 0x25f966 (0x69e4 bytes) 000447[1] 1789497692.030392 mapwrite 0x1c1f8f thru 0x1c79e4 (0x5a56 bytes) 000449[0] 1789497692.151944 trunc from 0x985636 to 0x52631f (0x45f316 bytes) 000450[0] 1789497692.195814 write 0x798904 thru 0x7a568a (0xcd87 bytes) 000451[1] 1789497692.234353 read 0x2fff2a thru 0x3098d3 (0x99aa bytes) 000452[0] 1789497692.265839 trunc from 0x7a568b to 0x06c6a9 (0x738fe1 bytes) 000453[0] 1789497692.321196 mapwrite 0x4c74ef thru 0x4cb733 (0x4245 bytes) 000455[0] 1789497692.404856 mapread 0x0d51ba thru 0x0d9079 (0x3ec0 bytes) 000456[0] 1789497692.429893 write 0x467e35 thru 0x46e382 (0x654e bytes) 000458[0] 1789497692.564195 trunc from 0x4cb734 to 0x097cb5 (0x433a7e bytes) 000459[0] 1789497692.613037 mapwrite 0x61fdfc thru 0x62e372 (0xe577 bytes) 000460[0] 1789497692.716721 mapwrite 0x3e902f thru 0x3eca90 (0x3a62 bytes) 000461[1] 1789497692.786692 trunc from 0x62e373 to 0x1c98cb (0x464aa7 bytes) 000463[0] 1789497692.839545 write 0x74b553 thru 0x751a44 (0x64f2 bytes) 000464[1] 1789497692.867938 punch from 0x1b8016 to 0x1bcc3a (0x4c25 bytes) 000465[1] 1789497692.891292 read 0x2c9e62 thru 0x2cdfce (0x416d bytes) 000466[0] 1789497692.935073 mapwrite 0x274f1a thru 0x27b084 (0x616b bytes) 000467[1] 1789497693.093504 trunc from 0x751a45 to 0x5ae3c2 (0x1a3682 bytes) 000469[1] 1789497693.179688 trunc from 0x5ae3c3 to 0x8ef144 (0x340d82 bytes) 000470[1] 1789497693.208107 write 0x617035 thru 0x62692e (0xf8fa bytes) 000472[1] 1789497693.240293 mapwrite 0x693731 thru 0x6a345c (0xfd2c bytes) 000473[1] 1789497693.366341 read 0x8ce1d6 thru 0x8da7c5 (0xc5f0 bytes) 000475[0] 1789497693.422458 mapwrite 0x407b9a thru 0x41075f (0x8bc6 bytes) 000476[1] 1789497693.591550 punch from 0x44aa2c to 0x44f58f (0x4b64 bytes) 000477[0] 1789497693.605520 read 0x2d8f5e thru 0x2e8646 (0xf6e9 bytes) 000479[1] 1789497693.702054 punch from 0x3360c6 to 0x33e2c6 (0x8201 bytes) 000482[0] 1789497693.762892 mapwrite 0x218fd7 thru 0x21a826 (0x1850 bytes) 000484[0] 1789497693.915214 mapread 0x839a4d thru 0x842e21 (0x93d5 bytes) 000486[0] 1789497694.021023 mapread 0x5a86d6 thru 0x5a89af (0x2da bytes) 000487[0] 1789497694.085535 mapwrite 0x45ccf2 thru 0x46757f (0xa88e bytes) 000488[0] 1789497694.306387 mapread 0x2e730e thru 0x2f172b (0xa41e bytes) 000491[0] 1789497694.440812 read 0x581ad2 thru 0x58b687 (0x9bb6 bytes) 000492[1] 1789497694.468451 trunc from 0x8ef145 to 0x26086c (0x68e8d8 bytes) 000493[1] 1789497694.541302 mapread 0x0a8fc8 thru 0x0aef95 (0x5fce bytes) 000498[0] 1789497694.718964 punch from 0x24d558 to 0x2516a1 (0x414a bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7694 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x9cf8b6 000001[1] 1789497695.391545 trunc from 00000000 to 0x9cf8b5 (0x9cf8b6 bytes) 000002[1] 1789497695.409960 mapwrite 0x22c000 thru 0x22e9e9 (0x29ea bytes) 000007[0] 1789497695.607255 punch from 0x0ae5cf to 0x0b56ef (0x7121 bytes) 000008[1] 1789497695.645249 write 0x12a000 thru 0x12ef5f (0x4f60 bytes) 000014[1] 1789497695.740675 write 0x71d000 thru 0x71df46 (0xf47 bytes) 000015[1] 1789497695.745283 mapread 0x23a000 thru 0x243601 (0x9602 bytes) 000016[1] 1789497695.767672 trunc from 0x9cf8b6 to 0x993341 (0x3c574 bytes) 000017[0] 1789497695.811950 read 0x7ef000 thru 0x7f7555 (0x8556 bytes) 000018[0] 1789497695.842699 mapwrite 0x15c000 thru 0x16516d (0x916e bytes) 000021[0] 1789497695.942363 trunc from 0x993342 to 0x3da226 (0x5b911b bytes) 000023[1] 1789497695.974454 mapread 0x035000 thru 0x03bcb2 (0x6cb3 bytes) 000027[0] 1789497696.011374 mapwrite 0x6f1000 thru 0x6fc748 (0xb749 bytes) truncating to largest ever: 0x9f9071 000028[0] 1789497696.117888 trunc from 0x6fc749 to 0x9f9070 (0x2fc928 bytes) 000030[1] 1789497696.198841 mapread 0x37c000 thru 0x380755 (0x4756 bytes) 000031[0] 1789497696.254866 mapread 0x023000 thru 0x025002 (0x2003 bytes) 000032[0] 1789497696.289344 read 0x79f000 thru 0x7a3c59 (0x4c5a bytes) 000033[1] 1789497696.297917 write 0x08f000 thru 0x095e88 (0x6e89 bytes) 000034[0] 1789497696.316354 trunc from 0x9f9071 to 0x205a84 (0x7f35ec bytes) 000035[0] 1789497696.352731 mapread 0x044000 thru 0x047fb3 (0x3fb4 bytes) 000037[1] 1789497696.404074 read 0x089000 thru 0x08a83a (0x183b bytes) 000039[0] 1789497696.426980 write 0x22c000 thru 0x23a8c1 (0xe8c2 bytes) 000040[0] 1789497696.452996 trunc from 0x23a8c2 to 0x42bcae (0x1f13ed bytes) 000041[0] 1789497696.494201 write 0x178000 thru 0x181444 (0x9445 bytes) 000043[1] 1789497696.526855 read 0x159000 thru 0x15db7d (0x4b7e bytes) 000047[0] 1789497696.564918 punch from 0x1da6d5 to 0x1e534c (0xac78 bytes) 000048[0] 1789497696.611949 read 0x016000 thru 0x01d5b3 (0x75b4 bytes) 000049[0] 1789497696.624453 trunc from 0x42bcaf to 0x883f86 (0x4582d8 bytes) 000050[1] 1789497696.667073 read 0x338000 thru 0x33829b (0x29c bytes) 000051[0] 1789497696.698521 read 0x177000 thru 0x1792e4 (0x22e5 bytes) 000053[1] 1789497696.704855 punch from 0x119e5a to 0x11a865 (0xa0c bytes) 000056[1] 1789497696.765916 punch from 0x3a95da to 0x3b6040 (0xca67 bytes) 000057[1] 1789497696.773660 mapwrite 0x865000 thru 0x86ab67 (0x5b68 bytes) 000059[1] 1789497696.922840 read 0x81a000 thru 0x82548b (0xb48c bytes) 000060[0] 1789497696.945350 write 0x070000 thru 0x076e4b (0x6e4c bytes) 000061[1] 1789497696.976419 mapread 0x5e5000 thru 0x5f38f2 (0xe8f3 bytes) 000062[0] 1789497697.064020 read 0x3f6000 thru 0x404173 (0xe174 bytes) 000063[1] 1789497697.077382 read 0x038000 thru 0x03dd01 (0x5d02 bytes) 000064[1] 1789497697.088513 punch from 0x5e3652 to 0x5f21f6 (0xeba5 bytes) 000067[0] 1789497697.138882 write 0x402000 thru 0x411539 (0xf53a bytes) 000070[0] 1789497697.226697 read 0x67d000 thru 0x686732 (0x9733 bytes) 000071[1] 1789497697.258695 trunc from 0x883f87 to 0x2fab2e (0x589458 bytes) 000073[1] 1789497697.338369 mapwrite 0x01b000 thru 0x024fbd (0x9fbe bytes) 000074[0] 1789497697.436253 mapwrite 0x518000 thru 0x5226a6 (0xa6a7 bytes) 000075[1] 1789497697.568318 write 0x5c7000 thru 0x5d6b2e (0xfb2f bytes) 000078[0] 1789497697.627455 mapwrite 0x259000 thru 0x267ad3 (0xead4 bytes) 000081[0] 1789497697.789441 write 0x9d9000 thru 0x9db958 (0x2959 bytes) 000085[0] 1789497697.841733 punch from 0x8bbdcc to 0x8c5089 (0x92be bytes) 000086[0] 1789497697.875431 trunc from 0x9db959 to 0x961e4d (0x79b0b bytes) 000087[1] 1789497697.918868 write 0x365000 thru 0x373a87 (0xea88 bytes) 000088[1] 1789497697.942320 mapwrite 0x2a4000 thru 0x2ad2eb (0x92ec bytes) 000091[1] 1789497698.105043 write 0x89c000 thru 0x8a2cf7 (0x6cf8 bytes) 000092[1] 1789497698.114956 mapwrite 0x4a1000 thru 0x4a88da (0x78db bytes) 000093[1] 1789497698.248880 write 0x95a000 thru 0x9666b6 (0xc6b7 bytes) 000094[0] 1789497698.250405 punch from 0x8875c8 to 0x892b68 (0xb5a1 bytes) 000097[0] 1789497698.398034 read 0x081000 thru 0x08cb1d (0xbb1e bytes) 000098[0] 1789497698.451932 write 0x3a7000 thru 0x3b6cf5 (0xfcf6 bytes) 000099[1] 1789497698.470887 write 0x45f000 thru 0x462913 (0x3914 bytes) 000100[0] 1789497698.533934 mapread 0x42c000 thru 0x434e1b (0x8e1c bytes) 000101[0] 1789497698.576701 write 0x987000 thru 0x993230 (0xc231 bytes) 000102[0] 1789497698.590968 mapwrite 0x721000 thru 0x72fecd (0xeece bytes) 000103[1] 1789497698.710885 write 0x379000 thru 0x3875a9 (0xe5aa bytes) 000104[1] 1789497698.747995 write 0x236000 thru 0x23a5c1 (0x45c2 bytes) 000105[1] 1789497698.772603 mapread 0x376000 thru 0x384b6c (0xeb6d bytes) 000106[0] 1789497698.807524 mapwrite 0x26f000 thru 0x279082 (0xa083 bytes) 000108[1] 1789497698.985608 write 0x3ed000 thru 0x3efea9 (0x2eaa bytes) 000109[0] 1789497699.002353 punch from 0x7d5957 to 0x7e2b89 (0xd233 bytes) 000110[1] 1789497699.023676 read 0x5b9000 thru 0x5c6f28 (0xdf29 bytes) 000112[1] 1789497699.045458 read 0x367000 thru 0x36d460 (0x6461 bytes) 000113[1] 1789497699.053646 write 0x94d000 thru 0x94ff0a (0x2f0b bytes) 000115[1] 1789497699.071865 write 0x39b000 thru 0x3a1f45 (0x6f46 bytes) 000116[0] 1789497699.076992 read 0x46b000 thru 0x470da0 (0x5da1 bytes) 000117[1] 1789497699.103856 mapread 0x197000 thru 0x1a6be0 (0xfbe1 bytes) 000119[0] 1789497699.132437 trunc from 0x993231 to 0x7f9241 (0x199fef bytes) 000120[0] 1789497699.166765 mapread 0x09f000 thru 0x0a1f09 (0x2f0a bytes) 000121[1] 1789497699.198868 mapread 0x567000 thru 0x571be9 (0xabea bytes) 000122[1] 1789497699.242288 punch from 0x6d83be to 0x6de861 (0x64a4 bytes) 000124[0] 1789497699.262335 mapwrite 0x813000 thru 0x8189a7 (0x59a8 bytes) 000127[1] 1789497699.369811 write 0x532000 thru 0x53ad26 (0x8d27 bytes) 000131[1] 1789497699.446751 trunc from 0x8189a8 to 0x95d47b (0x144ad4 bytes) 000133[0] 1789497699.504044 mapwrite 0x675000 thru 0x67a6dc (0x56dd bytes) 000135[1] 1789497699.631712 write 0x8d7000 thru 0x8dde5a (0x6e5b bytes) 000136[1] 1789497699.647834 write 0x231000 thru 0x23bc2d (0xac2e bytes) 000143[0] 1789497699.684887 write 0x09c000 thru 0x0a213f (0x6140 bytes) 000144[0] 1789497699.744550 trunc from 0x95d47c to 0x472493 (0x4eafe8 bytes) 000145[0] 1789497699.778523 write 0x734000 thru 0x73f645 (0xb646 bytes) 000146[0] 1789497699.806586 read 0x2ff000 thru 0x301322 (0x2323 bytes) 000147[1] 1789497699.812820 trunc from 0x73f646 to 0x92ac66 (0x1eb621 bytes) 000148[1] 1789497699.859690 mapread 0x119000 thru 0x11af7e (0x1f7f bytes) 000149[0] 1789497699.878895 punch from 0x3b77b2 to 0x3b7cfa (0x549 bytes) 000151[0] 1789497699.946963 mapread 0x274000 thru 0x279fe8 (0x5fe9 bytes) 000152[0] 1789497700.041425 write 0x972000 thru 0x97ef56 (0xcf57 bytes) 000154[0] 1789497700.087288 trunc from 0x97ef57 to 0x1db5cc (0x7a398a bytes) 000156[0] 1789497700.115892 mapwrite 0x8bb000 thru 0x8c6e88 (0xbe89 bytes) 000157[1] 1789497700.283847 trunc from 0x8c6e89 to 0x022ddd (0x8a40ab bytes) 000158[0] 1789497700.340632 punch from 0x016ba8 to 0x0181fd (0x1656 bytes) 000162[0] 1789497700.396832 trunc from 0x022dde to 0x8e3f04 (0x8c1127 bytes) 000164[1] 1789497700.480681 mapread 0x68f000 thru 0x6914be (0x24bf bytes) 000167[1] 1789497700.586419 mapwrite 0x4c1000 thru 0x4c2b4a (0x1b4b bytes) 000168[1] 1789497700.766511 trunc from 0x8e3f05 to 0x291711 (0x6527f3 bytes) 000169[0] 1789497700.848514 mapread 0x04d000 thru 0x054b0d (0x7b0e bytes) 000170[0] 1789497700.925548 punch from 0x2567dc to 0x25d9b0 (0x71d5 bytes) 000172[1] 1789497700.955385 trunc from 0x291712 to 0x085466 (0x20c2ab bytes) 000173[0] 1789497701.071396 punch from 0x01908c to 0x0225be (0x9533 bytes) 000174[0] 1789497701.101746 mapread 00000000 thru 0x005d76 (0x5d77 bytes) 000175[1] 1789497701.111864 read 0x004000 thru 0x00a9d6 (0x69d7 bytes) 000176[1] 1789497701.140437 read 0x079000 thru 0x079f54 (0xf55 bytes) 000178[0] 1789497701.144680 read 0x07e000 thru 0x0833f4 (0x53f5 bytes) 000179[0] 1789497701.160022 mapwrite 0x168000 thru 0x172df3 (0xadf4 bytes) 000181[1] 1789497701.302264 read 0x0b3000 thru 0x0bdcb2 (0xacb3 bytes) 000182[1] 1789497701.359395 read 0x008000 thru 0x016649 (0xe64a bytes) 000184[1] 1789497701.365014 trunc from 0x172df4 to 0x825ab2 (0x6b2cbf bytes) 000187[0] 1789497701.470170 punch from 0x19b3dd to 0x1a09b1 (0x55d5 bytes) 000188[1] 1789497701.489453 write 0x6c7000 thru 0x6c7044 (0x45 bytes) 000189[0] 1789497701.582367 punch from 0x6149ba to 0x61ed18 (0xa35f bytes) 000191[0] 1789497701.671655 trunc from 0x825ab3 to 0x02e33e (0x7f7774 bytes) 000192[1] 1789497701.714923 mapread 0x00a000 thru 0x00ccbb (0x2cbc bytes) 000195[1] 1789497701.806646 punch from 0x026b53 to 0x02e33e (0x77ec bytes) 000197[0] 1789497701.887917 punch from 0x02c661 to 0x02e33e (0x1cde bytes) 000198[0] 1789497701.969686 mapread 0x026000 thru 0x02db6c (0x7b6d bytes) 000199[1] 1789497702.019246 trunc from 0x02e33f to 0x851db9 (0x823a7b bytes) 000204[0] 1789497702.099051 read 0x5ed000 thru 0x5fa7dd (0xd7de bytes) 000207[1] 1789497702.120390 write 0x17e000 thru 0x18b720 (0xd721 bytes) 000208[1] 1789497702.139092 trunc from 0x851dba to 0x87ac90 (0x28ed7 bytes) 000209[1] 1789497702.161631 mapread 0x5d2000 thru 0x5e19be (0xf9bf bytes) 000211[1] 1789497702.268035 punch from 0x83807a to 0x83c704 (0x468b bytes) 000212[1] 1789497702.279242 read 0x791000 thru 0x7a0c50 (0xfc51 bytes) 000215[0] 1789497702.291866 mapread 0x252000 thru 0x256c6e (0x4c6f bytes) 000217[0] 1789497702.363357 punch from 0x58c8f2 to 0x59155d (0x4c6c bytes) 000219[0] 1789497702.411687 mapread 0x533000 thru 0x53a779 (0x777a bytes) 000221[0] 1789497702.441893 trunc from 0x87ac91 to 0x36fb3b (0x50b155 bytes) 000222[1] 1789497702.458354 read 0x1b9000 thru 0x1c70f7 (0xe0f8 bytes) 000223[0] 1789497702.479858 mapread 0x03f000 thru 0x04bab8 (0xcab9 bytes) 000224[1] 1789497702.511075 mapread 0x16b000 thru 0x16e650 (0x3651 bytes) 000225[0] 1789497702.537951 read 0x30e000 thru 0x311f91 (0x3f92 bytes) 000226[0] 1789497702.598182 punch from 0x29d609 to 0x2a837b (0xad73 bytes) 000227[0] 1789497702.619976 punch from 0x093729 to 0x09b59a (0x7e72 bytes) 000229[0] 1789497702.626217 trunc from 0x36fb3c to 0x08e0b2 (0x2e1a89 bytes) 000230[1] 1789497702.652495 write 0x03c000 thru 0x03f710 (0x3711 bytes) 000232[1] 1789497702.691082 trunc from 0x08e0b3 to 0x9af00c (0x920f5a bytes) 000234[1] 1789497702.739074 trunc from 0x9af00d to 0x1469e2 (0x86862a bytes) 000235[1] 1789497702.767468 write 0x77d000 thru 0x781ead (0x4eae bytes) 000236[0] 1789497702.781392 trunc from 0x781eae to 0x6fd400 (0x84aad bytes) 000237[1] 1789497702.836395 trunc from 0x6fd401 to 0x46f88a (0x28db76 bytes) 000238[0] 1789497702.857948 trunc from 0x46f88b to 0x9d799f (0x568115 bytes) 000239[0] 1789497702.885795 punch from 0x5dd459 to 0x5ecfcc (0xfb74 bytes) 000240[1] 1789497702.912979 trunc from 0x9d79a0 to 0x0600f0 (0x9778af bytes) 000241[0] 1789497702.941689 mapread 0x03c000 thru 0x040566 (0x4567 bytes) 000243[1] 1789497702.982667 mapread 0x025000 thru 0x02d540 (0x8541 bytes) 000244[0] 1789497702.996627 trunc from 0x0600f1 to 0x45d9eb (0x3fd8fb bytes) 000245[1] 1789497703.052878 mapwrite 0x343000 thru 0x346270 (0x3271 bytes) 000247[1] 1789497703.194072 write 0x4a6000 thru 0x4ac7ec (0x67ed bytes) 000248[0] 1789497703.212338 mapread 0x28e000 thru 0x2996f8 (0xb6f9 bytes) 000252[1] 1789497703.325845 write 0x4ab000 thru 0x4b52fd (0xa2fe bytes) 000254[1] 1789497703.364909 punch from 0x3d5d33 to 0x3de7b8 (0x8a86 bytes) 000256[1] 1789497703.381737 trunc from 0x4b52fe to 0x3dbfa1 (0xd935c bytes) 000257[1] 1789497703.401105 trunc from 0x3dbfa2 to 0x2fe0cd (0xdded4 bytes) 000258[0] 1789497703.423421 write 0x967000 thru 0x975572 (0xe573 bytes) 000260[0] 1789497703.456799 read 0x22a000 thru 0x236e4b (0xce4c bytes) 000261[0] 1789497703.463574 punch from 0x13469b to 0x1419fb (0xd361 bytes) 000263[1] 1789497703.520809 write 0x872000 thru 0x875b91 (0x3b92 bytes) 000267[0] 1789497703.588731 read 0x861000 thru 0x86754a (0x654b bytes) 000268[1] 1789497703.646092 mapwrite 0x5fd000 thru 0x603eb2 (0x6eb3 bytes) 000269[0] 1789497703.797450 mapread 0x35e000 thru 0x362b83 (0x4b84 bytes) 000275[1] 1789497703.856502 write 0x46c000 thru 0x47959c (0xd59d bytes) 000277[0] 1789497703.909017 mapwrite 0x65e000 thru 0x667c68 (0x9c69 bytes) 000280[1] 1789497704.031718 punch from 0x6df111 to 0x6e88be (0x97ae bytes) 000281[1] 1789497704.048850 punch from 0x8fae81 to 0x8fd0d0 (0x2250 bytes) 000282[0] 1789497704.053069 write 0x78b000 thru 0x793fdb (0x8fdc bytes) 000283[1] 1789497704.079484 trunc from 0x975573 to 0x7c95c1 (0x1abfb1 bytes) 000284[1] 1789497704.164824 write 0x23d000 thru 0x240e88 (0x3e89 bytes) 000285[1] 1789497704.172606 read 0x07a000 thru 0x088eac (0xeead bytes) 000288[1] 1789497704.195864 trunc from 0x7c95c2 to 0x828b5a (0x5f599 bytes) 000290[1] 1789497704.292669 trunc from 0x828b5b to 0x2def37 (0x549c23 bytes) 000291[0] 1789497704.320862 mapread 0x235000 thru 0x238ba7 (0x3ba8 bytes) 000294[1] 1789497704.397000 mapwrite 0x881000 thru 0x883bd7 (0x2bd8 bytes) 000295[0] 1789497704.578826 write 0x255000 thru 0x26363d (0xe63e bytes) 000296[0] 1789497704.625010 mapwrite 0x8d9000 thru 0x8def4c (0x5f4d bytes) 000297[1] 1789497704.727622 punch from 0x3c134d to 0x3cee90 (0xdb44 bytes) 000298[0] 1789497704.748023 punch from 0x68328f to 0x6891f7 (0x5f69 bytes) 000301[1] 1789497704.796974 read 0x1f7000 thru 0x1ff3ef (0x83f0 bytes) 000303[1] 1789497704.813798 mapwrite 0x20b000 thru 0x20d53b (0x253c bytes) 000304[0] 1789497704.882271 read 0x548000 thru 0x555748 (0xd749 bytes) 000307[1] 1789497704.914951 write 0x9eb000 thru 0x9ec539 (0x153a bytes) 000308[0] 1789497704.930930 mapwrite 0x50e000 thru 0x51c08e (0xe08f bytes) 000310[1] 1789497705.051056 read 0x6bc000 thru 0x6bca5f (0xa60 bytes) 000312[0] 1789497705.080338 trunc from 0x9ec53a to 0x8909c1 (0x15bb78 bytes) 000314[0] 1789497705.144362 mapwrite 0x06d000 thru 0x074ed8 (0x7ed9 bytes) 000315[0] 1789497705.290755 mapwrite 0x71a000 thru 0x7201d5 (0x61d6 bytes) 000316[0] 1789497705.428231 trunc from 0x8909c2 to 0x53d84c (0x353175 bytes) 000317[0] 1789497705.467226 write 0x5c9000 thru 0x5c946e (0x46f bytes) 000318[1] 1789497705.474337 mapwrite 0x26c000 thru 0x273db2 (0x7db3 bytes) 000322[0] 1789497705.631671 write 0x7ef000 thru 0x7ef260 (0x261 bytes) 000323[0] 1789497705.652674 read 0x1fe000 thru 0x209a81 (0xba82 bytes) 000325[0] 1789497705.684535 punch from 0x0989e2 to 0x09c8f6 (0x3f15 bytes) 000326[1] 1789497705.691634 mapread 0x0d5000 thru 0x0ddbb1 (0x8bb2 bytes) 000327[0] 1789497705.744420 read 0x157000 thru 0x15cfbe (0x5fbf bytes) 000328[0] 1789497705.750776 write 0x6a2000 thru 0x6a7547 (0x5548 bytes) 000330[1] 1789497705.774741 read 0x0f2000 thru 0x100a15 (0xea16 bytes) 000332[1] 1789497705.797688 trunc from 0x7ef261 to 0x2ad047 (0x542219 bytes) 000335[0] 1789497705.879394 punch from 0x092dbb to 0x096adb (0x3d21 bytes) 000337[1] 1789497705.925292 read 0x0af000 thru 0x0bb226 (0xc227 bytes) 000338[1] 1789497705.967565 write 0x53b000 thru 0x53d56c (0x256d bytes) 000339[1] 1789497706.005427 mapread 0x13d000 thru 0x14cddc (0xfddd bytes) 000341[0] 1789497706.205457 punch from 0x29a41f to 0x2a58cb (0xb4ad bytes) 000342[1] 1789497706.238224 mapwrite 0x367000 thru 0x36711c (0x11d bytes) 000343[1] 1789497706.342990 read 0x1e4000 thru 0x1f2547 (0xe548 bytes) 000344[1] 1789497706.353574 trunc from 0x53d56d to 0x4a320d (0x9a35f bytes) 000346[1] 1789497706.378020 read 0x1b6000 thru 0x1b754a (0x154b bytes) 000347[0] 1789497706.396618 write 0x420000 thru 0x42a5cb (0xa5cc bytes) 000348[0] 1789497706.410649 trunc from 0x4a320e to 0x38cace (0x11673f bytes) 000351[1] 1789497706.446651 mapwrite 0x1c6000 thru 0x1cdc6c (0x7c6d bytes) 000352[1] 1789497706.565269 punch from 0x1775ce to 0x184854 (0xd287 bytes) 000354[1] 1789497706.602870 mapread 0x37d000 thru 0x38310a (0x610b bytes) 000355[0] 1789497706.675372 mapwrite 0x705000 thru 0x70a737 (0x5738 bytes) 000356[1] 1789497706.780677 trunc from 0x70a738 to 0x7b262f (0xa7ef8 bytes) 000357[0] 1789497706.835529 mapread 0x2eb000 thru 0x2f5387 (0xa388 bytes) 000360[1] 1789497706.889036 mapwrite 0x84f000 thru 0x859ac4 (0xaac5 bytes) 000363[1] 1789497707.020568 trunc from 0x859ac5 to 0x85e87e (0x4dba bytes) 000364[1] 1789497707.098409 mapread 0x34a000 thru 0x34c45b (0x245c bytes) 000366[1] 1789497707.160874 trunc from 0x85e87f to 0x45355e (0x40b320 bytes) 000368[0] 1789497707.216581 mapread 0x066000 thru 0x066ba9 (0xbaa bytes) 000370[1] 1789497707.262730 write 0x319000 thru 0x31f5e7 (0x65e8 bytes) 000371[0] 1789497707.306683 write 0x723000 thru 0x72844b (0x544c bytes) 000372[1] 1789497707.368879 punch from 0x656181 to 0x65d151 (0x6fd1 bytes) 000375[0] 1789497707.433411 mapread 0x5a4000 thru 0x5ac794 (0x8795 bytes) 000376[0] 1789497707.482309 mapread 0x20a000 thru 0x20fc5e (0x5c5f bytes) 000380[0] 1789497707.511936 write 0x79c000 thru 0x7a8271 (0xc272 bytes) 000382[0] 1789497707.553437 write 0x249000 thru 0x24be88 (0x2e89 bytes) 000384[0] 1789497707.570578 trunc from 0x7a8272 to 0x825abb (0x7d84a bytes) 000387[1] 1789497707.623931 write 0x486000 thru 0x489c9c (0x3c9d bytes) 000388[0] 1789497707.658311 write 0x114000 thru 0x12169a (0xd69b bytes) 000389[0] 1789497707.685337 punch from 0x630baf to 0x638982 (0x7dd4 bytes) 000391[1] 1789497707.724814 read 0x0c5000 thru 0x0d3c06 (0xec07 bytes) 000392[0] 1789497707.775992 read 0x0d7000 thru 0x0e3592 (0xc593 bytes) 000393[0] 1789497707.783647 read 0x7ea000 thru 0x7f0b8b (0x6b8c bytes) 000395[0] 1789497707.800347 mapread 0x14a000 thru 0x150c6d (0x6c6e bytes) 000396[0] 1789497707.825600 write 0x060000 thru 0x060c6f (0xc70 bytes) 000397[1] 1789497707.858840 read 0x782000 thru 0x782490 (0x491 bytes) 000398[0] 1789497707.879566 trunc from 0x825abc to 0x260311 (0x5c57aa bytes) 000399[1] 1789497707.936855 punch from 0x131329 to 0x137ab5 (0x678d bytes) 000400[0] 1789497707.990632 punch from 0x085f30 to 0x08c925 (0x69f6 bytes) 000401[1] 1789497708.019672 mapwrite 0x055000 thru 0x056d78 (0x1d79 bytes) 000402[0] 1789497708.126364 mapwrite 0x201000 thru 0x20a5c1 (0x95c2 bytes) 000406[0] 1789497708.238544 mapwrite 0x688000 thru 0x68db6b (0x5b6c bytes) 000408[1] 1789497708.342209 mapread 0x031000 thru 0x03acfe (0x9cff bytes) 000409[1] 1789497708.382878 trunc from 0x68db6c to 0x2aa45e (0x3e370d bytes) 000410[1] 1789497708.427586 write 0x658000 thru 0x664887 (0xc888 bytes) 000411[1] 1789497708.437276 mapread 0x541000 thru 0x541c37 (0xc38 bytes) 000412[1] 1789497708.444990 read 0x51f000 thru 0x527cc5 (0x8cc6 bytes) 000413[1] 1789497708.476031 trunc from 0x664888 to 0x4de60c (0x18627b bytes) 000414[1] 1789497708.507776 read 0x1ad000 thru 0x1b1585 (0x4586 bytes) 000417[1] 1789497708.592017 mapwrite 0x461000 thru 0x461304 (0x305 bytes) 000419[1] 1789497708.696698 punch from 0x24d060 to 0x257953 (0xa8f4 bytes) 000420[0] 1789497708.707344 mapwrite 0x14e000 thru 0x1597ee (0xb7ef bytes) 000421[0] 1789497708.875663 trunc from 0x4de60d to 0x997543 (0x4b8f37 bytes) 000422[0] 1789497708.931736 trunc from 0x997544 to 0x52c036 (0x46b50d bytes) 000423[1] 1789497708.959232 trunc from 0x52c037 to 0x5fb4a6 (0xcf470 bytes) 000424[1] 1789497709.003625 mapwrite 0x87b000 thru 0x884801 (0x9802 bytes) 000425[0] 1789497709.099724 mapread 0x5e2000 thru 0x5ea3cf (0x83d0 bytes) 000426[1] 1789497709.147427 read 0x27d000 thru 0x28c5a2 (0xf5a3 bytes) 000427[0] 1789497709.165412 read 0x7d3000 thru 0x7d7957 (0x4958 bytes) 000430[0] 1789497709.175357 trunc from 0x884802 to 0x0980b9 (0x7ec748 bytes) 000431[1] 1789497709.238736 punch from 0x043a34 to 0x0444ca (0xa97 bytes) 000432[0] 1789497709.269317 mapread 0x027000 thru 0x02af59 (0x3f5a bytes) 000433[0] 1789497709.323701 write 0x809000 thru 0x80f729 (0x672a bytes) 000437[1] 1789497709.408896 mapread 0x78c000 thru 0x796620 (0xa621 bytes) 000438[1] 1789497709.450435 read 0x494000 thru 0x498dd3 (0x4dd4 bytes) 000439[0] 1789497709.461395 mapread 0x519000 thru 0x51f900 (0x6901 bytes) 000441[0] 1789497709.483825 read 0x0a3000 thru 0x0a882e (0x582f bytes) 000443[0] 1789497709.490735 mapwrite 0x314000 thru 0x31701a (0x301b bytes) 000445[1] 1789497709.585849 mapread 0x74f000 thru 0x753332 (0x4333 bytes) 000446[1] 1789497709.626335 trunc from 0x80f72a to 0x0a4d8d (0x76a99c bytes) 000451[1] 1789497709.706743 punch from 0x063b26 to 0x067635 (0x3b10 bytes) 000454[0] 1789497709.739679 punch from 0x092954 to 0x09afc6 (0x8673 bytes) 000456[1] 1789497709.777758 write 0x186000 thru 0x192fde (0xcfdf bytes) 000457[0] 1789497709.792779 mapread 0x105000 thru 0x109731 (0x4732 bytes) 000460[0] 1789497709.843643 punch from 0x03bb8e to 0x049f47 (0xe3ba bytes) 000461[1] 1789497709.869868 write 0x0c4000 thru 0x0c453a (0x53b bytes) 000462[1] 1789497709.911984 punch from 0x046bcd to 0x04ad30 (0x4164 bytes) 000464[1] 1789497709.952886 mapread 0x003000 thru 0x00fb6e (0xcb6f bytes) 000465[1] 1789497709.975900 write 0x924000 thru 0x92d716 (0x9717 bytes) 000466[1] 1789497709.993066 mapwrite 0x73e000 thru 0x744dc2 (0x6dc3 bytes) 000467[0] 1789497710.072243 write 0x63e000 thru 0x64d771 (0xf772 bytes) 000469[1] 1789497710.086471 mapwrite 0x712000 thru 0x71feaa (0xdeab bytes) 000470[1] 1789497710.158905 write 0x717000 thru 0x71b5ad (0x45ae bytes) 000471[1] 1789497710.169000 punch from 0x24b075 to 0x24d56d (0x24f9 bytes) 000472[0] 1789497710.179812 punch from 0x87561e to 0x87bdc4 (0x67a7 bytes) 000473[1] 1789497710.228040 read 0x78d000 thru 0x79178b (0x478c bytes) 000474[1] 1789497710.255860 write 0x1ba000 thru 0x1c8639 (0xe63a bytes) 000476[0] 1789497710.300811 read 0x314000 thru 0x31819b (0x419c bytes) 000478[1] 1789497710.348641 punch from 0x3b1e80 to 0x3bc63f (0xa7c0 bytes) 000481[0] 1789497710.473748 trunc from 0x92d717 to 0x085be9 (0x8a7b2d bytes) 000482[0] 1789497710.528959 punch from 0x07b077 to 0x08000f (0x4f99 bytes) 000484[0] 1789497710.552169 mapwrite 0x5bf000 thru 0x5c6b28 (0x7b29 bytes) 000485[1] 1789497710.673843 mapwrite 0x2c1000 thru 0x2d07bb (0xf7bc bytes) 000486[0] 1789497710.825213 read 0x4b1000 thru 0x4b46a7 (0x36a8 bytes) 000488[0] 1789497710.858868 read 0x10a000 thru 0x118d8f (0xed90 bytes) 000489[0] 1789497710.871891 punch from 0x2c9ccc to 0x2d6fd9 (0xd30e bytes) 000491[1] 1789497710.907494 trunc from 0x5c6b29 to 0x2e8475 (0x2de6b3 bytes) 000494[1] 1789497710.970897 read 0x260000 thru 0x26231e (0x231f bytes) 000495[1] 1789497711.025306 read 0x2df000 thru 0x2e7ad2 (0x8ad3 bytes) 000496[0] 1789497711.025883 read 0x113000 thru 0x1143a7 (0x13a8 bytes) 000497[0] 1789497711.100670 mapread 0x15e000 thru 0x169268 (0xb269 bytes) 000498[1] 1789497711.101390 punch from 0x0a8047 to 0x0a888f (0x849 bytes) 000500[0] 1789497711.134786 trunc from 0x2e8476 to 0x2bf05b (0x2941a bytes) All operations completed A-OK! PASS 16a (49s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:41:55 (1789497715) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.379858 s, 22.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.248395 s, 33.8 MB/s Chance of close/open is 1 in 50 Seed set to 7718 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000007[0] 1789497719.404188 write 0x36dc1d thru 0x36f85f (0x1c43 bytes) 000008[1] 1789497719.407905 mapread 0x16cc52 thru 0x16db66 (0xf15 bytes) 000009[1] 1789497719.490723 punch from 0x1eadab to 0x1eb2e4 (0x53a bytes) 000012[0] 1789497719.651658 write 0x3bbfc6 thru 0x3bd0f0 (0x112b bytes) 000015[1] 1789497719.708858 write 0x95515a thru 0x957018 (0x1ebf bytes) 000016[0] 1789497719.757350 mapread 0x5e34bb thru 0x5e49a3 (0x14e9 bytes) 000018[1] 1789497719.823554 read 0x5ad327 thru 0x5ade31 (0xb0b bytes) 000019[1] 1789497719.858480 mapwrite 0x867ce1 thru 0x8697e3 (0x1b03 bytes) truncating to largest ever: 0x6be64a 000021[0] 1789497719.948678 trunc from 0x957019 to 0x6be649 (0x2989cf bytes) 000022[0] 1789497719.978434 punch from 0x67acad to 0x67adc5 (0x119 bytes) 000023[0] 1789497719.997985 trunc from 0x6be64a to 0x55fd8f (0x15e8ba bytes) 000024[0] 1789497720.028947 mapread 0x448779 thru 0x44a1f9 (0x1a81 bytes) 000025[0] 1789497720.084559 mapread 0x505ca0 thru 0x506086 (0x3e7 bytes) 000026[1] 1789497720.117099 punch from 0x2cebbc to 0x2cf694 (0xad9 bytes) 000027[0] 1789497720.145839 write 0x3aa8f1 thru 0x3aabd8 (0x2e8 bytes) 000032[0] 1789497720.195265 read 0x52f599 thru 0x52ff7b (0x9e3 bytes) 000033[1] 1789497720.218941 mapwrite 0x71bb95 thru 0x71d2e2 (0x174e bytes) 000036[1] 1789497720.401727 mapread 0x45aded thru 0x45ccc4 (0x1ed8 bytes) 000039[0] 1789497720.529269 write 0x5ec7d7 thru 0x5ed294 (0xabe bytes) 000040[1] 1789497720.555903 read 0x4b1640 thru 0x4b1898 (0x259 bytes) 000041[0] 1789497720.580576 write 0x756c70 thru 0x75825b (0x15ec bytes) 000042[0] 1789497720.600426 read 0x5809bc thru 0x581a7f (0x10c4 bytes) 000043[0] 1789497720.614538 mapwrite 0x80556a thru 0x806305 (0xd9c bytes) 000045[1] 1789497720.710982 mapwrite 0x60b023 thru 0x60be2d (0xe0b bytes) 000047[0] 1789497720.869107 trunc from 0x806306 to 0x3f4fff (0x411306 bytes) 000048[0] 1789497720.922093 mapread 0x136b09 thru 0x13758c (0xa84 bytes) 000049[1] 1789497720.951174 write 0x270d3d thru 0x272624 (0x18e8 bytes) 000051[0] 1789497721.077309 mapread 0x387626 thru 0x3891b8 (0x1b93 bytes) 000053[1] 1789497721.194205 read 0x3d0946 thru 0x3d1b09 (0x11c4 bytes) 000054[0] 1789497721.343955 mapwrite 0x58aace thru 0x58baa5 (0xfd8 bytes) 000055[1] 1789497721.486945 read 0x57c7cc thru 0x57d117 (0x94c bytes) 000056[1] 1789497721.560333 mapwrite 0x9c2f28 thru 0x9c2ff4 (0xcd bytes) 000057[0] 1789497721.768291 punch from 0x878260 to 0x8782a2 (0x43 bytes) 000059[1] 1789497721.838871 punch from 0x3df582 to 0x3e0014 (0xa93 bytes) 000061[1] 1789497721.853433 write 0x940a60 thru 0x9411a8 (0x749 bytes) 000062[0] 1789497722.047298 trunc from 0x9c2ff5 to 0x05074b (0x9728a9 bytes) 000064[0] 1789497722.260107 write 0x61136d thru 0x612b19 (0x17ad bytes) 000067[1] 1789497722.341294 punch from 0x2e76a1 to 0x2e8fe1 (0x1941 bytes) 000068[0] 1789497722.408977 write 0x07d54b thru 0x07d93f (0x3f5 bytes) 000070[0] 1789497722.514056 write 0x0fa93a thru 0x0fc80e (0x1ed5 bytes) 000073[0] 1789497722.651851 mapread 0x5e42c2 thru 0x5e5324 (0x1063 bytes) 000075[0] 1789497722.734424 read 0x07515a thru 0x075f5d (0xe04 bytes) 000076[1] 1789497722.743732 punch from 0x5e4136 to 0x5e60f9 (0x1fc4 bytes) 000078[1] 1789497722.840843 read 0x03c3a3 thru 0x03d074 (0xcd2 bytes) 000079[0] 1789497722.953649 mapread 0x3e4fc5 thru 0x3e6081 (0x10bd bytes) 000086[0] 1789497723.232827 mapread 0x4822a6 thru 0x4828af (0x60a bytes) 000087[0] 1789497723.269510 mapread 0x323ea7 thru 0x3245c4 (0x71e bytes) 000089[1] 1789497723.279554 mapread 0x276e1c thru 0x277bff (0xde4 bytes) 000092[1] 1789497723.288230 mapwrite 0x6d57ce thru 0x6d5914 (0x147 bytes) 000093[1] 1789497723.403921 mapread 0x5a2e9a thru 0x5a4824 (0x198b bytes) 000094[1] 1789497723.490254 read 0x0c7f46 thru 0x0c8433 (0x4ee bytes) 000095[1] 1789497723.511237 trunc from 0x6d5915 to 0x30c787 (0x3c918d bytes) 000096[0] 1789497723.592431 mapwrite 0x160b39 thru 0x160b3b (0x3 bytes) 000097[1] 1789497723.668414 write 0x9083ab thru 0x908426 (0x7c bytes) 000098[0] 1789497723.695044 mapread 0x001965 thru 0x0025a3 (0xc3f bytes) 000103[1] 1789497723.832094 write 0x11b9ea thru 0x11d6c4 (0x1cdb bytes) 000105[0] 1789497723.968177 mapread 0x08a21e thru 0x08afe0 (0xdc3 bytes) 000106[1] 1789497724.056843 punch from 0x7ff2dd to 0x800611 (0x1335 bytes) 000108[0] 1789497724.100697 mapwrite 0x52d33d thru 0x52f1b9 (0x1e7d bytes) 000109[1] 1789497724.317312 write 0x2e3ab9 thru 0x2e52eb (0x1833 bytes) 000111[0] 1789497724.361101 punch from 0x49773e to 0x498672 (0xf35 bytes) 000114[0] 1789497724.411056 read 0x488f00 thru 0x489919 (0xa1a bytes) 000115[0] 1789497724.438610 mapwrite 0x3cbfe3 thru 0x3cd2eb (0x1309 bytes) truncating to largest ever: 0x85ae98 000116[0] 1789497724.504306 trunc from 0x908427 to 0x85ae97 (0xad58f bytes) 000118[1] 1789497724.540882 write 0x5e1ea2 thru 0x5e3b6a (0x1cc9 bytes) 000119[1] 1789497724.565938 mapread 0x7dc1af thru 0x7dcda4 (0xbf6 bytes) 000121[0] 1789497724.615402 write 0x0fe290 thru 0x0feda2 (0xb13 bytes) 000124[1] 1789497724.656714 mapread 0x050655 thru 0x05105a (0xa06 bytes) 000125[1] 1789497724.722675 write 0x4b9748 thru 0x4bb698 (0x1f51 bytes) 000126[0] 1789497724.751239 mapread 0x8438f4 thru 0x8443bb (0xac8 bytes) 000128[0] 1789497724.816738 trunc from 0x85ae98 to 0x706a1c (0x15447b bytes) 000129[1] 1789497724.855044 trunc from 0x706a1d to 0x5abd02 (0x15ad1a bytes) truncating to largest ever: 0x88606a 000132[1] 1789497724.918172 trunc from 0x5abd03 to 0x886069 (0x2da367 bytes) 000133[1] 1789497724.967874 mapread 0x595925 thru 0x596cb8 (0x1394 bytes) 000137[1] 1789497725.031303 trunc from 0x88606a to 0x05dd5d (0x82830c bytes) 000138[1] 1789497725.064966 write 0x6e4cc1 thru 0x6e6953 (0x1c93 bytes) 000143[0] 1789497725.108371 trunc from 0x6e6954 to 0x28779d (0x45f1b6 bytes) 000144[1] 1789497725.185250 read 0x1ef51c thru 0x1f0752 (0x1237 bytes) 000145[0] 1789497725.237032 read 0x2707df thru 0x272098 (0x18ba bytes) 000146[1] 1789497725.306044 mapwrite 0x39fbc0 thru 0x3a137f (0x17c0 bytes) 000147[0] 1789497725.469615 mapread 0x11116e thru 0x112a4d (0x18e0 bytes) 000148[1] 1789497725.554616 mapread 0x34defe thru 0x34e6cd (0x7d0 bytes) 000150[0] 1789497725.606303 read 0x26858c thru 0x268dee (0x863 bytes) 000152[1] 1789497725.689687 read 0x34eebb thru 0x350588 (0x16ce bytes) 000154[1] 1789497725.768668 punch from 0x344bad to 0x34612d (0x1581 bytes) 000156[0] 1789497725.816062 punch from 0x123ca2 to 0x125b44 (0x1ea3 bytes) 000157[0] 1789497725.831944 write 0x7e9dfa thru 0x7ea89b (0xaa2 bytes) 000158[0] 1789497725.843302 mapwrite 0x95bf91 thru 0x95d1c0 (0x1230 bytes) 000160[0] 1789497726.035692 punch from 0x3a0429 to 0x3a0648 (0x220 bytes) 000162[0] 1789497726.041585 punch from 0x21c56a to 0x21dc38 (0x16cf bytes) 000163[1] 1789497726.044929 write 0x17789d thru 0x177907 (0x6b bytes) 000164[1] 1789497726.077469 punch from 0x8f41cc to 0x8f5e4f (0x1c84 bytes) 000167[1] 1789497726.123496 mapread 0x94f26b thru 0x94f52f (0x2c5 bytes) 000168[0] 1789497726.146682 trunc from 0x95d1c1 to 0x702d00 (0x25a4c0 bytes) 000170[0] 1789497726.185345 mapread 0x5a202b thru 0x5a3fa7 (0x1f7d bytes) 000171[0] 1789497726.207495 mapwrite 0x32fcf6 thru 0x33118f (0x149a bytes) 000172[1] 1789497726.286609 write 0x4e368b thru 0x4e4e95 (0x180b bytes) 000174[1] 1789497726.424856 punch from 0x6b32a4 to 0x6b4563 (0x12c0 bytes) 000176[1] 1789497726.467933 read 0x138664 thru 0x138e6a (0x807 bytes) 000177[1] 1789497726.523733 punch from 0x1c7e31 to 0x1c8780 (0x950 bytes) 000178[0] 1789497726.535676 mapread 0x2f098a thru 0x2f253a (0x1bb1 bytes) 000180[0] 1789497726.614653 punch from 0x0adacb to 0x0ae8bc (0xdf2 bytes) 000181[1] 1789497726.667434 write 0x8b825d thru 0x8b9b1d (0x18c1 bytes) 000182[1] 1789497726.682894 write 0x6e9202 thru 0x6ea95f (0x175e bytes) 000184[1] 1789497726.771247 mapwrite 0x614d58 thru 0x6154cb (0x774 bytes) 000186[1] 1789497726.869571 trunc from 0x8b9b1e to 0x7ea958 (0xcf1c5 bytes) 000187[1] 1789497726.936647 mapwrite 0x5d4e99 thru 0x5d6b5d (0x1cc5 bytes) 000191[0] 1789497727.047172 mapwrite 0x0c27f4 thru 0x0c39f1 (0x11fe bytes) 000194[1] 1789497727.150454 mapwrite 0x3280ac thru 0x328782 (0x6d7 bytes) 000197[0] 1789497727.288443 mapread 0x559bc5 thru 0x55b924 (0x1d60 bytes) 000198[0] 1789497727.357492 write 0x01ca23 thru 0x01cec6 (0x4a4 bytes) 000200[0] 1789497727.393301 punch from 0x0e0088 to 0x0e1eac (0x1e25 bytes) 000201[0] 1789497727.404709 punch from 0x2450fe to 0x2468f6 (0x17f9 bytes) 000202[0] 1789497727.413820 mapread 0x6e9bbe thru 0x6e9bd1 (0x14 bytes) 000204[1] 1789497727.419255 mapwrite 0x812577 thru 0x813259 (0xce3 bytes) 000208[1] 1789497727.562019 trunc from 0x81325a to 0x1beaed (0x65476c bytes) 000209[0] 1789497727.640487 mapread 0x0e7ad9 thru 0x0e7dd2 (0x2fa bytes) 000211[1] 1789497727.718080 mapwrite 0x07cff9 thru 0x07ec4a (0x1c52 bytes) 000212[1] 1789497727.843400 mapread 0x15e954 thru 0x1602fd (0x19aa bytes) 000213[0] 1789497727.850305 mapwrite 0x9e04cb thru 0x9e1b56 (0x168c bytes) 000214[1] 1789497727.961908 mapwrite 0x922076 thru 0x922c20 (0xbab bytes) 000215[0] 1789497728.196023 punch from 0x9c20eb to 0x9c3dd9 (0x1cef bytes) 000217[1] 1789497728.305306 read 0x5d9f63 thru 0x5dbd6b (0x1e09 bytes) 000218[1] 1789497728.352633 mapread 0x094f27 thru 0x095b0f (0xbe9 bytes) 000221[1] 1789497728.386120 punch from 0x88cbf0 to 0x88d2fc (0x70d bytes) 000224[0] 1789497728.421104 mapread 0x4846c7 thru 0x485fe5 (0x191f bytes) 000225[0] 1789497728.433738 mapwrite 0x603217 thru 0x603463 (0x24d bytes) 000226[0] 1789497728.486723 punch from 0x72d535 to 0x72f45a (0x1f26 bytes) 000227[1] 1789497728.498879 write 0x5a85f1 thru 0x5a9a54 (0x1464 bytes) 000228[0] 1789497728.530418 mapread 0x1bc3cd thru 0x1bca14 (0x648 bytes) 000229[1] 1789497728.569487 read 0x38e5c7 thru 0x39038b (0x1dc5 bytes) 000230[0] 1789497728.576044 trunc from 0x9e1b57 to 0x47d195 (0x5649c1 bytes) 000231[0] 1789497728.632669 trunc from 0x47d196 to 0x13576e (0x347a27 bytes) 000232[0] 1789497728.647811 mapwrite 0x80417a thru 0x8049fe (0x885 bytes) 000233[0] 1789497728.753031 write 0x955483 thru 0x955c4c (0x7ca bytes) 000235[0] 1789497728.790576 write 0x340228 thru 0x341f6d (0x1d46 bytes) 000236[0] 1789497728.825880 mapread 0x7aad67 thru 0x7abacf (0xd69 bytes) 000237[1] 1789497728.855201 mapwrite 0x3abc4e thru 0x3ad7f4 (0x1ba7 bytes) 000239[0] 1789497729.013890 trunc from 0x955c4d to 0x27c691 (0x6d95bb bytes) 000240[0] 1789497729.043384 punch from 0x12cca6 to 0x12e539 (0x1894 bytes) 000241[0] 1789497729.048392 mapwrite 0x28c10b thru 0x28cbc5 (0xabb bytes) 000242[0] 1789497729.200884 punch from 0x03eba3 to 0x0403a6 (0x1804 bytes) 000243[0] 1789497729.223340 mapread 0x26ad9c thru 0x26aec1 (0x126 bytes) 000245[1] 1789497729.236647 mapwrite 0x213999 thru 0x2153ab (0x1a13 bytes) 000247[0] 1789497729.369768 mapread 0x02f531 thru 0x031185 (0x1c55 bytes) 000249[1] 1789497729.421025 write 0x62e28e thru 0x62ed1a (0xa8d bytes) 000250[0] 1789497729.474707 write 0x99d654 thru 0x99f45d (0x1e0a bytes) truncating to largest ever: 0x8fc827 000251[0] 1789497729.556261 trunc from 0x99f45e to 0x8fc826 (0xa2c37 bytes) 000252[0] 1789497729.630744 punch from 0x6fd8e7 to 0x6fe46e (0xb88 bytes) 000254[1] 1789497729.669182 read 0x434aa7 thru 0x43637c (0x18d6 bytes) 000255[1] 1789497729.704395 punch from 0x458fa3 to 0x459dbc (0xe1a bytes) 000256[1] 1789497729.729222 mapread 0x72c015 thru 0x72dc15 (0x1c01 bytes) 000258[1] 1789497729.782224 write 0x14ac48 thru 0x14b489 (0x842 bytes) 000261[0] 1789497729.829986 read 0x5016e2 thru 0x503689 (0x1fa8 bytes) 000263[0] 1789497729.855036 trunc from 0x8fc827 to 0x161198 (0x79b68e bytes) 000265[1] 1789497729.948632 mapread 0x0173e6 thru 0x018e42 (0x1a5d bytes) 000266[0] 1789497730.002575 read 0x04751f thru 0x048f37 (0x1a19 bytes) 000267[1] 1789497730.053375 write 0x9c4409 thru 0x9c5e6e (0x1a66 bytes) 000268[1] 1789497730.088254 write 0x86237e thru 0x8632b1 (0xf34 bytes) 000271[0] 1789497730.196210 mapwrite 0x506fa8 thru 0x507e89 (0xee2 bytes) truncating to largest ever: 0x90fa60 000272[1] 1789497730.415456 trunc from 0x9c5e6f to 0x90fa5f (0xb640f bytes) 000275[1] 1789497730.599183 write 0x9b330e thru 0x9b4753 (0x1446 bytes) 000277[1] 1789497730.638825 trunc from 0x9b4754 to 0x6d3863 (0x2e0ef0 bytes) 000279[1] 1789497730.679110 write 0x1024bc thru 0x102587 (0xcc bytes) 000280[0] 1789497730.714530 mapwrite 0x877340 thru 0x879186 (0x1e47 bytes) 000281[0] 1789497731.191463 mapwrite 0x60b237 thru 0x60cd12 (0x1adc bytes) 000285[1] 1789497731.327752 mapread 0x0faae6 thru 0x0fb486 (0x9a1 bytes) 000286[0] 1789497731.398200 trunc from 0x879187 to 0x19821a (0x6e0f6c bytes) 000287[0] 1789497731.487110 mapwrite 0x4d94e9 thru 0x4d9ed4 (0x9ec bytes) 000288[1] 1789497731.612263 trunc from 0x4d9ed5 to 0x4db58a (0x16b6 bytes) 000289[0] 1789497731.705670 punch from 0x28f8c3 to 0x290d55 (0x1493 bytes) 000290[0] 1789497731.735996 mapread 0x208812 thru 0x20a650 (0x1e3f bytes) 000292[0] 1789497731.776678 mapread 0x32fba6 thru 0x330a95 (0xef0 bytes) 000294[0] 1789497731.841313 trunc from 0x4db58b to 0x77cc1f (0x2a1695 bytes) 000295[1] 1789497731.877924 mapwrite 0x717bcb thru 0x718e87 (0x12bd bytes) 000296[1] 1789497731.964123 write 0x8a6f67 thru 0x8a7856 (0x8f0 bytes) 000297[1] 1789497731.972066 punch from 0x4eed95 to 0x4ef1b4 (0x420 bytes) 000298[1] 1789497731.975339 read 0x4404ac thru 0x441b15 (0x166a bytes) 000302[1] 1789497732.028389 write 0x12b19f thru 0x12c03b (0xe9d bytes) 000303[1] 1789497732.041389 write 0x55f7f3 thru 0x55ff16 (0x724 bytes) 000305[1] 1789497732.129197 punch from 0x1fff84 to 0x201f01 (0x1f7e bytes) 000308[1] 1789497732.224768 punch from 0x1fc90a to 0x1fd768 (0xe5f bytes) 000309[1] 1789497732.261659 mapwrite 0x094678 thru 0x094955 (0x2de bytes) 000311[0] 1789497732.357365 trunc from 0x8a7857 to 0x081c5c (0x825bfa bytes) 000312[0] 1789497732.454849 punch from 0x03b5b4 to 0x03c066 (0xab3 bytes) 000313[0] 1789497732.484526 mapwrite 0x0a2800 thru 0x0a46cc (0x1ecd bytes) 000317[0] 1789497732.655622 punch from 0x007277 to 0x008531 (0x12bb bytes) 000318[1] 1789497732.704601 read 0x004092 thru 0x00460c (0x57b bytes) 000322[0] 1789497732.794344 punch from 0x00c37b to 0x00db8e (0x1814 bytes) 000324[1] 1789497732.842714 read 0x02b0ec thru 0x02b24b (0x160 bytes) 000325[1] 1789497732.903072 trunc from 0x0a46cd to 0x7be7fb (0x71a12f bytes) 000326[0] 1789497732.946294 mapwrite 0x5b6dc0 thru 0x5b89f3 (0x1c34 bytes) 000328[1] 1789497733.085074 read 0x5b1953 thru 0x5b31e9 (0x1897 bytes) 000330[0] 1789497733.110729 trunc from 0x7be7fc to 0x0a0695 (0x71e166 bytes) 000331[1] 1789497733.208943 read 0x05ba0b thru 0x05d752 (0x1d48 bytes) 000332[0] 1789497733.284402 mapread 0x024cae thru 0x0263b4 (0x1707 bytes) 000333[1] 1789497733.353014 mapread 0x02916c thru 0x029433 (0x2c8 bytes) 000338[1] 1789497733.398186 mapread 0x053fbf thru 0x0548a5 (0x8e7 bytes) 000342[0] 1789497733.497667 write 0x046973 thru 0x046a5a (0xe8 bytes) 000345[0] 1789497733.548880 read 0x00cc95 thru 0x00ea73 (0x1ddf bytes) 000349[0] 1789497733.617871 trunc from 0x0a0696 to 0x7d8760 (0x7380cb bytes) 000352[0] 1789497733.637345 punch from 0x37c1ed to 0x37d3a5 (0x11b9 bytes) 000354[1] 1789497733.661041 mapread 0x00b767 thru 0x00ca4e (0x12e8 bytes) 000355[0] 1789497733.741561 write 0x3f5156 thru 0x3f5c46 (0xaf1 bytes) 000356[0] 1789497733.837880 mapwrite 0x6d9cca thru 0x6dab18 (0xe4f bytes) 000357[1] 1789497733.983755 trunc from 0x7d8761 to 0x421916 (0x3b6e4a bytes) 000358[1] 1789497734.072779 mapread 0x021cd2 thru 0x022567 (0x896 bytes) 000359[1] 1789497734.130389 write 0x73907f thru 0x73918f (0x111 bytes) 000360[0] 1789497734.138576 punch from 0x606957 to 0x607a52 (0x10fc bytes) 000361[0] 1789497734.192674 mapread 0x1c49ac thru 0x1c511f (0x774 bytes) 000362[0] 1789497734.232879 read 0x72d9d5 thru 0x72e819 (0xe45 bytes) 000365[0] 1789497734.271823 mapwrite 0x6f5ae3 thru 0x6f780e (0x1d2c bytes) 000370[0] 1789497734.466583 punch from 0x2f777b to 0x2f7f63 (0x7e9 bytes) 000372[0] 1789497734.485242 mapread 0x53e1ed thru 0x53fb2d (0x1941 bytes) 000373[0] 1789497734.497235 mapwrite 0x634896 thru 0x634c28 (0x393 bytes) 000375[0] 1789497734.607188 write 0x89b4d3 thru 0x89bb7d (0x6ab bytes) 000376[0] 1789497734.609492 trunc from 0x89bb7e to 0x359aca (0x5420b3 bytes) 000378[1] 1789497734.652245 write 0x16c45d thru 0x16c67d (0x221 bytes) 000379[0] 1789497734.694450 read 0x1f2238 thru 0x1f249f (0x268 bytes) 000380[0] 1789497734.803438 read 0x0bec38 thru 0x0bee15 (0x1de bytes) 000382[1] 1789497734.850449 mapwrite 0x353930 thru 0x354ffe (0x16cf bytes) 000383[0] 1789497734.936363 write 0x4d5e3d thru 0x4d6c96 (0xe5a bytes) 000385[1] 1789497734.986113 trunc from 0x4d6c97 to 0x28e100 (0x248b96 bytes) 000388[0] 1789497735.082719 write 0x003aa6 thru 0x004fed (0x1548 bytes) 000389[1] 1789497735.216904 trunc from 0x28e101 to 0x160d6a (0x12d396 bytes) 000391[1] 1789497735.317582 mapread 0x0d3289 thru 0x0d48ea (0x1662 bytes) 000397[1] 1789497735.512156 trunc from 0x160d6b to 0x176f2b (0x161c1 bytes) 000399[1] 1789497735.631720 write 0x2ffbdb thru 0x2ffefe (0x324 bytes) 000401[1] 1789497735.653649 read 0x25f788 thru 0x25fad7 (0x350 bytes) 000402[0] 1789497735.725097 write 0x30a03f thru 0x30a2ce (0x290 bytes) 000404[0] 1789497735.779631 write 0x0bffca thru 0x0c0514 (0x54b bytes) 000405[1] 1789497735.828997 trunc from 0x30a2cf to 0x667700 (0x35d432 bytes) 000406[0] 1789497735.907206 write 0x518eca thru 0x519fa6 (0x10dd bytes) truncating to largest ever: 0x9871e5 000407[0] 1789497735.943239 trunc from 0x667701 to 0x9871e4 (0x31fae4 bytes) 000408[0] 1789497735.972394 punch from 0x32850c to 0x32883f (0x334 bytes) 000409[0] 1789497735.992601 trunc from 0x9871e5 to 0x61b412 (0x36bdd2 bytes) 000412[0] 1789497736.030969 read 0x4216ec thru 0x421eee (0x803 bytes) 000413[0] 1789497736.057691 read 0x0cde98 thru 0x0cfe63 (0x1fcc bytes) 000415[0] 1789497736.114545 mapread 0x464f02 thru 0x466749 (0x1848 bytes) 000416[0] 1789497736.157011 punch from 0x2e7264 to 0x2e8f15 (0x1cb2 bytes) 000418[0] 1789497736.193162 write 0x467889 thru 0x467dc4 (0x53c bytes) 000419[0] 1789497736.193888 mapread 0x57833c thru 0x579f65 (0x1c2a bytes) 000420[0] 1789497736.221105 mapwrite 0x6492da thru 0x64a3b3 (0x10da bytes) 000422[1] 1789497736.296377 read 0x45f429 thru 0x461269 (0x1e41 bytes) 000424[0] 1789497736.346273 mapwrite 0x9d0e37 thru 0x9d1bda (0xda4 bytes) 000425[1] 1789497736.419530 mapread 0x3b97b9 thru 0x3b9809 (0x51 bytes) 000426[0] 1789497736.441711 trunc from 0x9d1bdb to 0x397ee7 (0x639cf3 bytes) 000428[1] 1789497736.502944 trunc from 0x397ee8 to 0x15f568 (0x23897f bytes) 000429[0] 1789497736.562231 mapread 0x05daf4 thru 0x05f8cb (0x1dd8 bytes) 000432[1] 1789497736.616606 trunc from 0x15f569 to 0x5ef87f (0x490317 bytes) 000433[1] 1789497736.677592 mapwrite 0x8e0647 thru 0x8e0aef (0x4a9 bytes) 000435[0] 1789497736.765963 punch from 0x490439 to 0x490f54 (0xb1c bytes) 000436[0] 1789497736.788718 punch from 0x643e1c to 0x645010 (0x11f5 bytes) 000438[0] 1789497736.812357 trunc from 0x8e0af0 to 0x662970 (0x27e17f bytes) 000439[1] 1789497736.842424 punch from 0x4dabd4 to 0x4db2e8 (0x715 bytes) 000440[0] 1789497736.863885 read 0x29f5e3 thru 0x29f670 (0x8e bytes) 000441[1] 1789497736.876173 punch from 0x351792 to 0x353348 (0x1bb7 bytes) 000442[1] 1789497736.893944 read 0x3a470c thru 0x3a54c8 (0xdbd bytes) 000444[0] 1789497736.925097 trunc from 0x662971 to 0x32bf49 (0x336a27 bytes) 000445[1] 1789497736.958321 trunc from 0x32bf4a to 0x1c8451 (0x163af8 bytes) 000446[0] 1789497736.994634 punch from 0x104291 to 0x1055ac (0x131c bytes) 000449[1] 1789497737.065615 trunc from 0x1c8452 to 0x508c50 (0x3407ff bytes) 000450[1] 1789497737.108978 read 0x3d843f thru 0x3d9db9 (0x197b bytes) 000451[0] 1789497737.155532 mapwrite 0x27ba25 thru 0x27d447 (0x1a23 bytes) 000452[1] 1789497737.274159 mapread 0x17689c thru 0x177ee9 (0x164e bytes) 000455[0] 1789497737.330040 trunc from 0x508c51 to 0x01cb00 (0x4ec150 bytes) 000456[0] 1789497737.365799 trunc from 0x01cb01 to 0x1ff8ba (0x1e2dba bytes) 000458[0] 1789497737.417104 read 0x1da456 thru 0x1db144 (0xcef bytes) 000461[1] 1789497737.472271 write 0x47232c thru 0x4730e6 (0xdbb bytes) 000463[0] 1789497737.508911 read 0x287c91 thru 0x288bde (0xf4e bytes) 000466[0] 1789497737.554021 mapread 0x030bab thru 0x032832 (0x1c88 bytes) 000467[1] 1789497737.571525 write 0x85264b thru 0x852910 (0x2c6 bytes) 000468[1] 1789497737.590917 read 0x75d46a thru 0x75de25 (0x9bc bytes) 000469[1] 1789497737.595004 write 0x89d0bd thru 0x89d624 (0x568 bytes) 000470[1] 1789497737.614705 mapread 0x7eb3bc thru 0x7eb6c9 (0x30e bytes) 000472[0] 1789497737.658694 mapread 0x3d75c8 thru 0x3d7a19 (0x452 bytes) 000473[1] 1789497737.720032 mapwrite 0x81fb8d thru 0x821227 (0x169b bytes) 000476[1] 1789497737.794691 write 0x43efaa thru 0x4407f9 (0x1850 bytes) 000477[1] 1789497737.818330 punch from 0x6da55b to 0x6da99f (0x445 bytes) 000478[1] 1789497737.840345 read 0x461ac7 thru 0x461b7a (0xb4 bytes) 000479[1] 1789497737.844594 write 0x366eca thru 0x368452 (0x1589 bytes) 000480[0] 1789497737.857770 write 0x20889d thru 0x20a56d (0x1cd1 bytes) 000481[0] 1789497737.889590 mapwrite 0x6a9d69 thru 0x6aa55e (0x7f6 bytes) 000483[0] 1789497737.973561 mapread 0x15428d thru 0x154a02 (0x776 bytes) 000484[0] 1789497737.978678 write 0x507e49 thru 0x5080ee (0x2a6 bytes) 000486[0] 1789497737.998353 trunc from 0x89d625 to 0x04eee9 (0x84e73b bytes) 000487[1] 1789497738.035018 punch from 0x0435bf to 0x0441a0 (0xbe2 bytes) 000490[1] 1789497738.084318 punch from 0x005463 to 0x005534 (0xd2 bytes) 000491[0] 1789497738.097268 read 0x023c93 thru 0x025881 (0x1bef bytes) 000492[0] 1789497738.134901 mapread 0x029d1b thru 0x02bcbf (0x1fa5 bytes) 000494[0] 1789497738.162621 punch from 0x0176e0 to 0x0187c6 (0x10e7 bytes) 000495[0] 1789497738.194081 punch from 0x022f58 to 0x024c19 (0x1cc2 bytes) 000496[1] 1789497738.217846 read 0x016658 thru 0x0182d5 (0x1c7e bytes) 000498[0] 1789497738.254460 mapwrite 0x58460a thru 0x585792 (0x1189 bytes) 000500[1] 1789497738.382746 trunc from 0x585793 to 0x485743 (0x10004f bytes) All operations completed A-OK! PASS 16b (28s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:42:23 (1789497743) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:42:28 (1789497748) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.437251 s, 19.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.395538 s, 21.2 MB/s ldlm.namespaces.MGC192.168.203.129@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9db3c32fd000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9db3ca0cb000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9db3ca0cb000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9db3ca0cb000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1293 s, 9.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.15143 s, 33.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.46533 s, 42.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.00502 s, 17.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.19825 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.26295 s, 24.6 MB/s PASS 16d (49s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:43:17 (1789497797) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.552 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239764 s, 4.4 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:43:26 (1789497806) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:43:54 (1789497834) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 14:44:20 (1789497860) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 14:44:28 (1789497868) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 14:44:35 (1789497875) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.28672 s, 20.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.3729 s, 19.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54444 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77214 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.31851 s, 11.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41534 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28686 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.30402 s, 20.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.66084 s, 15.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.22426 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.33272 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.18669 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41273 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.9566 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.09334 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.99055 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.47314 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.58246 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.89154 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:45:12 (1789497912) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 15 14:45:59 EDT 2026 + fsx 2 done: rc=0 Tue Sep 15 14:45:59 EDT 2026 + fsx 3 done: rc=0 Tue Sep 15 14:45:59 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Tue Sep 15 14:46:00 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Tue Sep 15 14:46:00 EDT 2026 PASS 16k (56s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:46:08 (1789497968) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 14:46:18 (1789497978) mmap test1: basic mmap operation (PASS, 0.11572s) mmap test2: MAP_PRIVATE not write back (PASS, 0.086372s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.4603s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3994s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.21841s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22633s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2701s) mmap test11: multi-thread shared mmap (PASS, 1.816s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1027s) PASS 18 (26s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:46:44 (1789498004) oleg329-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:46:48 (1789498008) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:46:54 (1789498014) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 14:47:01 (1789498021) now is 1789498023 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 new atime is 1789498084 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:48:11 (1789498091) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:48:18 (1789498098) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:48:24 (1789498104) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:48:30 (1789498110) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 14:48:34 (1789498114) utime: good mknod times 1789498114 <= 1789498115 <= 1789498115 for /mnt/lustre/f26a utime: good mknod times 1789498114 <= 1789498115 <= 1789498115 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 14:48:40 (1789498120) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:48:49 (1789498129) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-15 14:48:50.000000000 -0400 Birth: 2026-09-15 14:48:51.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-15 14:48:50.000000000 -0400 Birth: 2026-09-15 14:48:51.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:48:58 (1789498138) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:49:06 (1789498146) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:49:13 (1789498153) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.129@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 14:49:35 (1789498175) -rw-r--r-- 1 root root 0 Sep 15 14:49 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:49:45 (1789498185) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43151 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:49:53 (1789498193) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 14:50:01 (1789498201) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 14:50:04 (1789498204) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39558 s, 37.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3491 s, 38.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90799 s, 27.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.15596 s, 24.3 MB/s contention_seconds 4, contention_events 15 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.66599 s, 31.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90006 s, 27.6 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.23018 s, 42.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33631 s, 39.2 MB/s PASS 32c (28s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:50:34 (1789498234) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:50:37 (1789498237) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:50:40 (1789498240) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:50:44 (1789498244) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 14:50:46 (1789498246) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid in FULL state after 0 sec oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid in IDLE state after 0 sec oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid in IDLE state after 0 sec oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid in IDLE state after 0 sec oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9db3c32fd000.ost_server_uuid in IDLE state after 0 sec oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9db3c32fd000.ost_server_uuid in IDLE state after 0 sec PASS 34 (138s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:53:05 (1789498385) fail_loc=0x80000317 Race attempt 0 Wait for 50456 50468 for 60 sec... - open/close 681 (time 1789498402.66 total 10.04 last 67.83) - open/close 1369 (time 1789498412.67 total 20.05 last 68.72) - open/close 2105 (time 1789498422.69 total 30.07 last 73.49) - open/close 2895 (time 1789498432.70 total 40.08 last 78.90) - open/close 3637 (time 1789498442.71 total 50.09 last 74.12) total: 4000 open/close in 55.07 seconds: 72.64 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.129@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.203.129@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:54:17 (1789498457) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45198 s, 36.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50289 s, 34.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07022 s, 49.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40151 s, 37.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59566 s, 32.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46432 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7499776) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13765 s, 46.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7499776) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16141 s, 45.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26464 s, 41.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26212 s, 41.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15846 s, 45.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7493632) after_dd(7446528) after(7497728) PASS 36 (301s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:59:18 (1789498758) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 - create 1516 (time 1789498771.53 total 10.00 last 151.53) - create 2842 (time 1789498781.54 total 20.01 last 132.57) - create 4211 (time 1789498791.54 total 30.01 last 136.88) - create 5760 (time 1789498801.54 total 40.01 last 154.88) - create 7153 (time 1789498811.54 total 50.02 last 139.21) - create 8537 (time 1789498821.55 total 60.02 last 138.29) - create 9917 (time 1789498831.55 total 70.03 last 137.99) total: 10000 create in 70.57 seconds: 141.71 ops/second PASS 37 (81s) == sanityn test 39a: file mtime does not change after rename ========================================================== 15:00:39 (1789498839) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 15:00:46 (1789498846) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 15:00:54 (1789498854) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 15:01:01 (1789498861) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 15:01:09 (1789498869) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (15s) == sanityn test 40c: pdirops: link and others ======================================================================== 15:01:24 (1789498884) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 15:01:39 (1789498899) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 15:01:54 (1789498914) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (14s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 15:02:08 (1789498928) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 15:02:19 (1789498939) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 15:02:32 (1789498952) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 15:02:44 (1789498964) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 15:02:56 (1789498976) Conflict PASS 41e (10s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 15:03:07 (1789498987) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 15:03:18 (1789498998) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 15:03:29 (1789499009) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (11s) == sanityn test 41i: reint_open: create vs create ======== 15:03:40 (1789499020) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1009s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 15:20:29 (1789500029) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 15:20:41 (1789500041) open: File exists Conflict PASS 42b (11s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 15:20:52 (1789500052) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (11s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 15:21:04 (1789500064) Conflict PASS 42d (11s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 15:21:15 (1789500075) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (12s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 15:21:27 (1789500087) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 15:21:39 (1789500099) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 15:21:51 (1789500111) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 15:22:03 (1789500123) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 15:23:07 (1789500187) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 15:23:19 (1789500199) Conflict PASS 43c (11s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 15:23:30 (1789500210) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (11s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 15:23:41 (1789500221) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 15:23:55 (1789500235) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 15:24:09 (1789500249) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 15:24:21 (1789500261) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 15:24:34 (1789500274) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 15:24:38 (1789500278) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (113s) == sanityn test 43k: unlink vs create ==================== 15:26:31 (1789500391) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1063s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 15:44:14 (1789501454) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 15 15:44:22 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 15:44:27 (1789501467) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 15:44:41 (1789501481) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 15:44:53 (1789501493) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 15:45:06 (1789501506) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 15:45:20 (1789501520) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 15:45:34 (1789501534) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 15:45:47 (1789501547) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 15:46:01 (1789501561) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 15:46:04 (1789501564) PASS 45a (116s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 15:48:00 (1789501680) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 15:48:13 (1789501693) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 15:48:25 (1789501705) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 15:48:38 (1789501718) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 15:48:50 (1789501730) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 15:49:03 (1789501743) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 15:49:15 (1789501755) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 15:49:27 (1789501767) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 15:49:30 (1789501770) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1084s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 16:07:34 (1789502854) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 16:07:46 (1789502866) open: File exists Conflict PASS 46b (12s) == sanityn test 46c: pdirops: link vs link ======================================================================== 16:07:58 (1789502878) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 16:08:10 (1789502890) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 16:08:22 (1789502902) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 16:08:34 (1789502914) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 16:08:46 (1789502926) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 16:09:00 (1789502940) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 16:09:14 (1789502954) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 16:09:17 (1789502957) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 16:09:20 (1789502960) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 16:09:23 (1789502963) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 16:09:25 (1789502965) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 16:09:28 (1789502968) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 16:09:32 (1789502972) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 16:09:35 (1789502975) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 16:09:38 (1789502978) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.215266 s, 47.6 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 16:09:51 (1789502991) 0+1 records in 0+1 records out 158 bytes copied, 0.0327014 s, 4.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218566) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 16:09:59 (1789502999) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00695567 s, 147 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070214 s, 14.9 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 16:10:18 (1789503018) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0400062 s, 25.6 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 16:10:31 (1789503031) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205694 s, 51.0 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 16:10:39 (1789503039) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 16:10:49 (1789503049) llite.lustre-ffff9db3c32fd000.xattr_cache=1 llite.lustre-ffff9db3ca0cb000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:10:51.000000000 -0400 Modify: 2026-09-15 16:10:51.000000000 -0400 Change: 2026-09-15 16:10:51.000000000 -0400 Birth: 2026-09-15 16:10:51.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:10:57.000000000 -0400 Modify: 2026-09-15 16:10:58.000000000 -0400 Change: 2026-09-15 16:10:58.000000000 -0400 Birth: 2026-09-15 16:10:57.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:11:03.000000000 -0400 Modify: 2026-09-15 16:11:03.000000000 -0400 Change: 2026-09-15 16:11:03.000000000 -0400 Birth: 2026-09-15 16:11:03.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:11:03.000000000 -0400 Modify: 2026-09-15 16:11:04.000000000 -0400 Change: 2026-09-15 16:11:04.000000000 -0400 Birth: 2026-09-15 16:11:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:11:09.000000000 -0400 Modify: 2026-09-15 16:11:10.000000000 -0400 Change: 2026-09-15 16:11:10.000000000 -0400 Birth: 2026-09-15 16:11:09.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:11:09.000000000 -0400 Modify: 2026-09-15 16:11:10.000000000 -0400 Change: 2026-09-15 16:11:10.000000000 -0400 Birth: 2026-09-15 16:11:09.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 16:11:20 (1789503080) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 16:11:33 (1789503093) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 16:11:47 (1789503107) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 16:12:06 (1789503126) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 16:12:20 (1789503140) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 16:12:24 (1789503144) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 16:12:45 (1789503165) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 16:13:10 (1789503190) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789503191 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.53543 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (123s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 16:15:13 (1789503313) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789503317 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.10977 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (122s) == sanityn test 60: Verify data_version behaviour ======== 16:17:14 (1789503434) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 16:17:22 (1789503442) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 16:17:30 (1789503450) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 16:17:40 (1789503460) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00602307 s, 10.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 16:17:44 (1789503464) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0095687 s, 6.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 16:17:48 (1789503468) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 16:17:52 (1789503472) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0249349 s, 2.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 16:17:57 (1789503477) llite.lustre-ffff9db3c32fd000.xattr_cache=1 llite.lustre-ffff9db3ca0cb000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 16:18:06 (1789503486) llite.lustre-ffff9db3c32fd000.xattr_cache=1 llite.lustre-ffff9db3ca0cb000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 16:18:15 (1789503495) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.025707 s, 39.8 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0228841 s, 44.7 kB/s 231384: get lock2 231386: get lock1 231384: done 231384: get lock1 231386: done 231386: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231386: cannot get lock2: Resource deadlock avoided 231386: exit rc=0 231384: done 231384: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 16:18:26 (1789503506) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.062176 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364185 s, 28.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231967 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 16:18:38 (1789503518) opening files: ........ closing files: ........ PASS 76 (69s) == sanityn test 77a: check FIFO NRS policy =============== 16:19:47 (1789503587) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.416245 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104336 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10305 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522491 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149211 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124017 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15972 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316662 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16416 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038735 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228522 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174772 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31522 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351656 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233155 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253469 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362449 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986432 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161466 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10593 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18007 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160513 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127468 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113787 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122238 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965347 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115059 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111498 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105283 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146198 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113766 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122288 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125187 s, 8.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 16:19:58 (1789503598) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25833 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345714 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683513 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138883 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145666 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150481 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17624 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196153 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192534 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199865 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221247 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223275 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206797 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22809 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253539 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257734 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268698 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284678 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305271 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321606 s, 32.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103232 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255684 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630056 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629601 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105813 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961532 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189343 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126203 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735126 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291147 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286946 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995085 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238115 s, 44.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.58728 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359449 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568018 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186671 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222351 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151829 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235813 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072577 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280556 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128984 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240462 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212888 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30636 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303971 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3899 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406592 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308763 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769575 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388927 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415196 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853272 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328126 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235577 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288772 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435679 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155801 s, 67.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427579 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762476 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412814 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653267 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140404 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286775 s, 36.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 16:20:12 (1789503612) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.36805 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874811 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982579 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119573 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965952 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126649 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130934 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100716 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232788 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181228 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35657 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264753 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19445 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324717 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413019 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402317 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423735 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716283 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246707 s, 425 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127515 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661591 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148614 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133267 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671062 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976139 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682828 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060026 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335114 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308476 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316793 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317444 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236997 s, 44.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291332 s, 36.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.284717 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414222 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757791 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884205 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079541 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153167 s, 6.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183251 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13115 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209958 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212786 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254336 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230952 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249463 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244247 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232828 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198895 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297079 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526574 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690697 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130496 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704521 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534641 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100621 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486687 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625045 s, 16.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074144 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842668 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111984 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773107 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150715 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856892 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773532 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110352 s, 9.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.492456 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358321 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042497 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051115 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779731 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104044 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594191 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105289 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727795 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652137 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218448 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210773 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27062 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300527 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318038 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302156 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451654 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330201 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795779 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074643 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718711 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434309 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834914 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988979 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088073 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622408 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112044 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741167 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112171 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805883 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128372 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862753 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951578 s, 11.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (14s) == sanityn test 77d: check TRR nrs policy ================ 16:20:26 (1789503626) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.559401 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803054 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472562 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184968 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212688 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202526 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204219 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233615 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225887 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228738 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229578 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26863 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269643 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281739 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296408 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258697 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484614 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368461 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570355 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123104 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363037 s, 28.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070347 s, 14.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622258 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101501 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064032 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083753 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776156 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902227 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908302 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907019 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951848 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603575 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911945 s, 11.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.556102 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758686 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115404 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111926 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16474 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115263 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183114 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155331 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254329 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18917 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252185 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271808 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240769 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226939 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228932 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293367 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253234 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310588 s, 338 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376118 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373898 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494072 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00697986 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388095 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00700849 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206741 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902571 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211697 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896476 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16037 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122341 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277906 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22167 s, 4.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.651042 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458686 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722869 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816444 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143433 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988905 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814259 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712257 s, 14.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168615 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184647 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169011 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264381 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248951 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258348 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24419 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22926 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315739 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405565 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472798 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509488 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463565 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591851 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466074 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516527 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557741 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612579 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063869 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513732 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679632 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658447 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149729 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077275 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880742 s, 11.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 16:20:43 (1789503643) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.29@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.530122 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484739 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707461 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600481 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893992 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110204 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641005 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991184 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871528 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772207 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183321 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181422 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2259 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166139 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166943 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246746 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267187 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106323 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10744 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805663 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630769 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736148 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106961 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999207 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587026 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950802 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082957 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134789 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095306 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112515 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852742 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112847 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966378 s, 10.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65705 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101009 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14198 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271155 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304204 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223964 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333754 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235035 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419117 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289764 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376277 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35259 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291838 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442454 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3601 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498891 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328129 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436561 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275677 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27719 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242068 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132337 s, 79.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210887 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123209 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989688 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182896 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136247 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125439 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817193 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908237 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074771 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554542 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052628 s, 19.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.548535 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955872 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792795 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103739 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258051 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308997 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147528 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175995 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412921 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221595 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418749 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333042 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410061 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220494 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485257 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33396 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501842 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177338 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159356 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116749 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156496 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163579 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126739 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134828 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121583 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149625 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125539 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159238 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16622 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20357 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169561 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157516 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25416 s, 4.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.534808 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670014 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119007 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143198 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221958 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231938 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264449 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307568 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354978 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305817 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410686 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331148 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316037 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360752 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378306 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439769 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479545 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176163 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153845 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213821 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121169 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190642 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153244 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185099 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142867 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206678 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209792 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147348 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150604 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153829 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164688 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15682 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229013 s, 4.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:21:11 (1789503671) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.399406 s, 42.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415607 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100169 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825256 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123088 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109727 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984165 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151945 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456656 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121001 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24733 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256772 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242901 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291713 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287934 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24021 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312796 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198317 s, 52.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181394 s, 57.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519747 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698365 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298114 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596472 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539881 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468377 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770413 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415432 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536186 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548998 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565917 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465028 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060719 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180549 s, 58.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412089 s, 40.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651229 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054795 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958404 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109603 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149632 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20771 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105588 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168515 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261513 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18976 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211822 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200922 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215677 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234734 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205812 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260487 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415667 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504997 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581046 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00274708 s, 382 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639961 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707924 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437618 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045597 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406432 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362708 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390907 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11178 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348807 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285051 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177504 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247673 s, 42.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.515913 s, 32.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970685 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105011 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165234 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211431 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197541 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175316 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217902 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170377 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214806 s, 4.9 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203583 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269849 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260118 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26054 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186968 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264793 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318664 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00234015 s, 448 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105197 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108775 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178479 s, 5.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125208 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203067 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135606 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105369 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10333 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983652 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881018 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538817 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490258 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991914 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255618 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729849 s, 14.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.391445 s, 42.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596301 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920325 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140692 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215302 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176492 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188796 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239227 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254448 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230332 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260076 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255234 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270697 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316442 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245654 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304437 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355888 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462612 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332846 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602115 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290249 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753779 s, 13.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470319 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527164 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740996 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108691 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991376 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109547 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107142 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112582 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762497 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080761 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853746 s, 12.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (21s) == sanityn test 77g: Change TBF type directly ============ 16:21:32 (1789503692) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (10s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:21:42 (1789503702) oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 16:21:54 (1789503714) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:22:19 (1789503739) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499919 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827919 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111814 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105569 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218497 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25457 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197692 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278575 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277254 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302308 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227835 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238253 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254849 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277129 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2558 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262216 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328733 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541589 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558008 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078181 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886742 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140875 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079054 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792358 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096105 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790269 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161686 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166187 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964857 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834051 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927432 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939172 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100135 s, 10.5 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.88304 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7117 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:23:24 (1789503804) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32517 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590407 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09439 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103311 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138426 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177579 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125706 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107878 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870562 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978969 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154012 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406718 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255747 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128216 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208894 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217616 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259109 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718686 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06986 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609007 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685619 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812612 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407155 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898891 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512814 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103506 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717847 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830163 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076607 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686957 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104158 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792651 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878147 s, 11.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9539 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9585 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30099 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857482 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951967 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110122 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716391 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13284 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931824 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197943 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183715 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222443 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15243 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230504 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209188 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254367 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227161 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265821 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228642 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078981 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101829 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823892 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117866 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928514 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873995 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110509 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931722 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105772 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101374 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091606 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1359 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151929 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110802 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11472 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127598 s, 8.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9292 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0023 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:25:57 (1789503957) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26643 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807506 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243387 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113031 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116726 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134553 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132385 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153231 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158297 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176804 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224335 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224247 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216493 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242849 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278813 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313109 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327998 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526177 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053533 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605209 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813902 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756265 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725151 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627298 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779745 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976555 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860082 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119636 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870908 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902064 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967124 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955669 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101539 s, 10.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9972 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9463 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33594 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122265 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131928 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144607 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201753 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248984 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264214 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260522 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351477 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40342 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374814 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370546 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395556 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435095 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433621 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40115 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466472 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748417 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073225 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958397 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108897 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113065 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101723 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111609 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102724 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118358 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121729 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869027 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10581 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947978 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894912 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842727 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11461 s, 9.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0604 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9312 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (153s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:28:31 (1789504111) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.655201 s, 25.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056037 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113327 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109702 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128047 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169418 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133333 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156709 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242002 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263859 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316496 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270448 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294406 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288949 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407777 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361509 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40445 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802321 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107112 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107389 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107647 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847674 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241733 s, 434 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778573 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124012 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120219 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822844 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853472 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129561 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420267 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735975 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295641 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397888 s, 26.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.97277 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9389 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.29@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.29@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.601883 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952725 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806208 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107266 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133347 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13728 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132675 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17594 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186198 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21166 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210917 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266904 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28079 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283752 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376878 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376736 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33078 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503401 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463496 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373758 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359666 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546192 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676322 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554694 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347542 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778328 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668513 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502068 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641795 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786701 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467896 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673346 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107336 s, 9.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.86772 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96064 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.29@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.560226 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406948 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994464 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853876 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132542 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116843 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12408 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1005 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248562 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24344 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193482 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276184 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27656 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24443 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277437 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256489 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289874 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852692 s, 123 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602105 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018677 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739759 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018102 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307183 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040739 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408199 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781813 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795655 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419507 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383519 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518578 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451273 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469511 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469526 s, 22.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.89152 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27507 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.665708 s, 25.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847351 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145367 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159978 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189143 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136597 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189442 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184494 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162124 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266138 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293875 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30112 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355183 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239651 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28943 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27177 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408859 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219017 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335808 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101571 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041727 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996711 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126799 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818518 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841584 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121265 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621531 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824833 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103306 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791766 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850216 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907257 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271161 s, 38.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.133 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90903 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68507 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95684 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01465 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051633 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721003 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902248 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123039 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560197 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119595 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132168 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126109 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182196 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114575 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172372 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668688 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136326 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156709 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243488 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223907 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371759 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670226 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148663 s, 70.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030353 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057887 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460892 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115343 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122414 s, 85.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167796 s, 62.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601818 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779214 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219081 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813575 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037411 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157467 s, 66.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976944 s, 10.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0026 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5942 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571993 s, 29.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104981 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945469 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572338 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100364 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250872 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275955 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130977 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235595 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281723 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324555 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181025 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222507 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370336 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333494 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256294 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289232 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798314 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560393 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102827 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998274 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109107 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10964 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232 s, 452 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160898 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934326 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815056 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165022 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105242 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901145 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124305 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641654 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147264 s, 7.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.14074 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93614 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.55543 s, 30.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399781 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022564 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848715 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660853 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107769 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109047 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132707 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154357 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150198 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167897 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173187 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169582 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224037 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216431 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232891 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245793 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203124 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899872 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115193 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1052 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526855 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642899 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236936 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683471 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499104 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936951 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747357 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573684 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121905 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857385 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801982 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0054009 s, 194 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2629 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92515 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.76 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97636 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (441s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 16:35:51 (1789504551) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503852 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611678 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524962 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853597 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05151 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103868 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137559 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175174 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17326 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109285 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226034 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306534 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203596 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362175 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370218 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357577 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340166 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239625 s, 438 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305134 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802667 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613447 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675279 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953961 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729934 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106432 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526612 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117213 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850396 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895219 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095405 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777434 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134481 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104697 s, 10.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.72279 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90679 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (53s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 16:36:44 (1789504604) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.29@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.29@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.608646 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742957 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660066 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874773 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210194 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210207 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19809 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230691 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205488 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236691 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258343 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291809 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301929 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314891 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318606 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274955 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280604 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498721 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953714 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687378 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526411 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800626 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117256 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116589 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846632 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642837 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818966 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111493 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119967 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103019 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664169 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856519 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885411 s, 11.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.93552 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89349 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (51s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 16:37:35 (1789504655) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.29@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.634054 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878607 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675037 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115679 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103661 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249629 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281624 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324424 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192421 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30706 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318217 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319576 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327906 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225665 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287733 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337852 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285152 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c32fd000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c32fd000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367515 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949099 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748363 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101741 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643461 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884468 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100986 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547177 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631547 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663637 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670827 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739489 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788686 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728842 s, 14.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788774 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827104 s, 12.7 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.96425 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09723 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 16:38:23 (1789504703) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.943701 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773054 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529551 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192322 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150799 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20639 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215502 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231396 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246501 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244165 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247132 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253311 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244505 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31102 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254222 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29459 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299092 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113552 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186129 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506846 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089067 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100924 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345466 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399036 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454537 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698485 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461976 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552896 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962579 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012749 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680664 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822062 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788525 s, 13.3 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7367 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0373 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.6486 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91705 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 16:39:59 (1789504799) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.16314 s, 14.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423944 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107635 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100436 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137228 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244259 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227897 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177834 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241695 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254111 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224945 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239594 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329121 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318352 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264949 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279492 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359823 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267686 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535567 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572988 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694486 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739996 s, 14.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741945 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673719 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106478 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599663 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391889 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761088 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766026 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793006 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876228 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079399 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821531 s, 12.8 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7633 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0534 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (74s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:41:13 (1789504873) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.609567 s, 27.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667589 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07819 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743968 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129673 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140959 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194415 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234211 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252194 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218512 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284032 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260034 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249374 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297962 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255724 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314579 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329411 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033939 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727761 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0056671 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979718 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335399 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368009 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967126 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671227 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543691 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349057 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343812 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366388 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695614 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955186 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140604 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360786 s, 291 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.20941 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94661 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg329-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.702637 s, 23.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541099 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071813 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865956 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136319 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135392 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123871 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185184 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143264 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1113 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215275 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18826 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217267 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117754 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142573 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182898 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297753 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570182 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703932 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539935 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453844 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467396 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733716 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127542 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646259 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122386 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910982 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584232 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506524 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838924 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373315 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329569 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252961 s, 41.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9642 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8642 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6211 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92749 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (140s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 16:43:33 (1789505013) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg329-client.virtnet /mnt/lustre (opts:) Stopping client oleg329-client.virtnet /mnt/lustre opts: Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg329-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg329-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg329-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.29,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg329-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.29,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg329-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.29,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70305/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.29,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg329-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre Starting client oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre Started clients oleg329-client.virtnet: 192.168.203.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Started clients oleg329-client.virtnet: 192.168.203.129@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9db3c5dc9000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9db3c8384800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3c5dc9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3c8384800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469503 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681706 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12093 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156909 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180428 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109405 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249686 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312415 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341526 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237928 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354264 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29529 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191979 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273062 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327855 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40416 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324788 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c5dc9000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c5dc9000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453203 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41368 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410897 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427957 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443543 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431111 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457034 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447712 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454991 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477447 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493062 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460874 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46883 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503188 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526706 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53227 s, 2.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.65413 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17785 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.419987 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786913 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718303 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978833 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832481 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11898 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455322 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356676 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545229 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610515 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744741 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093784 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141656 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151303 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17309 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151116 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180618 s, 5.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9db3c5dc9000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217454 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684434 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029788 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855939 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358737 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781731 s, 13.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534739 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739893 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500444 s, 21.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421712 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876412 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597036 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681577 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598061 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633576 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661938 s, 15.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.24035 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92408 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (209s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 16:47:02 (1789505222) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 16:47:21 (1789505241) On MGS 192.168.203.129, active = nodemap.active=1 On MGS 192.168.203.129, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.727816 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608632 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978796 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112222 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191049 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169595 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240658 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280737 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24635 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288106 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277427 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23743 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217768 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268194 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255755 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29052 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330377 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c5dc9000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c5dc9000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408628 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375834 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394395 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374599 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471776 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439197 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446259 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37838 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38748 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452727 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40221 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463003 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502688 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448188 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436656 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450507 s, 2.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.29232 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06296 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.129, TBF.id = On MGS 192.168.203.129, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.544193 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709698 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099467 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183899 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181512 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176863 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2514 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269666 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284344 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290636 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26911 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292531 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29362 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249737 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291986 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292292 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326378 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102295 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638741 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569022 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706642 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463183 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720163 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058352 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058489 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615179 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535429 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686959 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627545 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678005 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953381 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668729 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700859 s, 15.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg329-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.26182 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.66503 s, 39.3 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg329-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 On MGS 192.168.203.129, active = nodemap.active=1 On MGS 192.168.203.129, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.510102 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496443 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840936 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659133 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104636 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23941 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215668 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239243 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277275 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238453 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221299 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296918 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240367 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291811 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297759 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232551 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29875 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747036 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871028 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504674 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334875 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249238 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045668 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377847 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410451 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105723 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520816 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908141 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517566 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058822 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620865 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128621 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670673 s, 15.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6081 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89693 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.129, TBF.id = On MGS 192.168.203.129, active = nodemap.active=0 PASS 77kj (171s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 16:50:13 (1789505413) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 16:50:23 (1789505423) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg329-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.76517 s, 278 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg329-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1086 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 16:51:21 (1789505481) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.474935 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854258 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871487 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147632 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214501 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233747 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274516 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233398 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252602 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292251 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31021 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326635 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324257 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350546 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326905 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368381 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9db3c5dc9000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9db3c5dc9000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462261 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911591 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253623 s, 413 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252583 s, 415 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246421 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161098 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118406 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223242 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110389 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101998 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921463 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897906 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182713 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783255 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633594 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585648 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542377 s, 19.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.19052 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29323 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57719 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606124 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827277 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761466 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920536 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105472 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102801 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123825 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111001 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119372 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134476 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130813 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161555 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181023 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185647 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18004 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163333 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090849 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357861 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037135 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533807 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04254 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465718 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504261 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521968 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124183 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445282 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100444 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123844 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615129 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715018 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745952 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143022 s, 7.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg329-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg329-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.6315 s, 15.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0661 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (87s) == sanityn test 77o: Changing rank should not panic ====== 16:52:48 (1789505568) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 16:53:00 (1789505580) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (111s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 16:54:51 (1789505691) mds.MDS.mdt.nrs_policies=tbf oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg329-client: oleg329-server: ssh exited with exit code 36 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 oleg329-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg329-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg329-client: oleg329-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (39s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 16:55:30 (1789505730) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273574 kill md_thread xargs PASS 77r (54s) == sanityn test 77s: Check TBF LRU shrinker ============== 16:56:25 (1789505785) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (68s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 16:57:32 (1789505852) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 16:57:42 (1789505862) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 16:57:44.000000000 -0400 Modify: 2026-09-15 16:57:44.000000000 -0400 Change: 2026-09-15 16:57:44.000000000 -0400 Birth: 2026-09-15 16:57:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 16:58:01 (1789505881) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 16:58:05 (1789505885) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 16:58:09 (1789505889) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 16:58:13 (1789505893) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 16:58:17 (1789505897) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 16:58:20 (1789505900) - open/close 775 (time 1789505916.58 total 10.02 last 77.34) - open/close 1551 (time 1789505926.59 total 20.03 last 77.50) - open/close 2354 (time 1789505936.60 total 30.04 last 80.22) total: 2500 open/close in 31.73 seconds: 78.79 ops/second snapshot_time 1789506031.292864667 secs.nsecs start_time 1789505904.414305961 secs.nsecs elapsed_time 126.878558706 secs.nsecs open 2505 samples [usecs] 112 17702 3732493 9501763127 close 2504 samples [usecs] 141 8089 915509 651935345 mknod 2501 samples [usecs] 599 17539 3278661 7604981977 unlink 2500 samples [usecs] 729 83695 5157821 34034255267 mkdir 2 samples [usecs] 1036 1586 2622 3588692 rmdir 2 samples [usecs] 40230 68542 108772 6316458664 rename 2500 samples [usecs] 466 58362 16531136 236012010642 getattr 3761 samples [usecs] 9 12321 301477 744856257 parallel_rename_file 2500 samples [usecs] 466 58362 16531136 236012010642 crossdir_rename 2500 samples [usecs] 466 58362 16531136 236012010642 readpage 10 samples [usecs] 3673 52756 157704 5284892392 PASS 81d (137s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 17:00:37 (1789506037) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 17:00:44 (1789506044) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 17:00:48 (1789506048) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 17:01:01 (1789506061) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789506063 Finishing test test1 at 1789506068 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 17:01:13 (1789506073) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 17:01:16 (1789506076) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 17:01:20 (1789506080) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 17:01:23 (1789506083) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5770 0x168a 0x240000400 1 5732 0x1664 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5733 0x1665 0x280000400 0 5771 0x168b 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 17:01:40 (1789506100) osc.lustre-OST0000-osc-ffff9db3c5dc9000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9db3c8384800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9db3c5dc9000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9db3c8384800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.236185 s, 34.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99674 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290358 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0845347 s, 48.5 kB/s osc.lustre-OST0000-osc-ffff9db3c5dc9000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9db3c8384800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3c5dc9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9db3c8384800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 17:01:56 (1789506116) llite.lustre-ffff9db3c5dc9000.fast_read=0 llite.lustre-ffff9db3c8384800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0635169 s, 33.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9db3c5dc9000.fast_read=1 llite.lustre-ffff9db3c8384800.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 17:02:16 (1789506136) llite.lustre-ffff9db3c5dc9000.fast_read=0 llite.lustre-ffff9db3c8384800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00411994 s, 3.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9db3c5dc9000.fast_read=1 llite.lustre-ffff9db3c8384800.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 17:02:28 (1789506148) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 17:02:32 (1789506152) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0133627 s, 307 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 17:02:38 (1789506158) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14966 s, 14.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 17:02:46 (1789506166) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159125 s, 13.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (6s) == sanityn test 100e: DoM: read on open and file size ==== 17:02:52 (1789506172) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00415322 s, 2.9 MB/s PASS 100e (6s) == sanityn test 101a: Discard DoM data on unlink ========= 17:02:58 (1789506178) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00763918 s, 536 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 17:03:05 (1789506185) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148443 s, 276 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 17:03:12 (1789506192) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0134709 s, 304 kB/s PASS 101c (7s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 17:03:19 (1789506199) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 17:03:26 (1789506206) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789506209 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789506212 Finishing test test23 at 1789506213 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 17:03:40 (1789506220) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 17:03:43 (1789506223) fail_loc=0x416 PASS 105 (30s) == sanityn test 106a: Verify the btime via statx() ======= 17:04:13 (1789506253) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 17:04:18 (1789506258) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0203464 s, 50.3 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 17:04:19.000000000 -0400 Modify: 2026-09-15 17:04:19.000000000 -0400 Change: 2026-09-15 17:04:19.000000000 -0400 Birth: 2026-09-15 17:04:19.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938183 -rw-r--r-- 1 0 0 1789506259 1789506259 1789506259 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 17:04:19.000000000 -0400 Modify: 2026-09-15 17:04:19.000000000 -0400 Change: 2026-09-15 17:04:19.000000000 -0400 Birth: 2026-09-15 17:04:19.000000000 -0400 1789506259 1024 11 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 17:04:27 (1789506267) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 17:04:35 (1789506275) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.269056 s, 39.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5740 0x166c 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 17:04:44 (1789506284) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333238 s, 31.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5780 0x1694 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 17:04:57 (1789506297) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00370282 s, 2.7 MB/s llite.lustre-ffff9db3c5dc9000.hybrid_io=0 llite.lustre-ffff9db3c8384800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.52185 s, 1.9 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 17:05:09 (1789506309) 192.168.203.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 192.168.203.129@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 PASS 109 (68s) == sanityn test 110: do not grant another lock on resend ========================================================== 17:06:17 (1789506377) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 17:06:18.000000000 -0400 Modify: 2026-09-15 17:06:18.000000000 -0400 Change: 2026-09-15 17:06:18.000000000 -0400 Birth: 2026-09-15 17:06:19.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg329-client.virtnet oleg329-client.virtnet PASS 110 (167s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 17:09:04 (1789506544) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 17:09:07 (1789506547) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 17:09:11 (1789506551) lustre-OST0000-osc-ffff9db3c736a800 active. lustre-OST0001-osc-ffff9db3c736a800 active. lustre-OST0000-osc-ffff9db3c736a800 active. lustre-OST0001-osc-ffff9db3c736a800 active. MGC192.168.203.129@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 17:09:18 (1789506558) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 17:09:21 (1789506561) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 17:09:24 (1789506564) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 17:09:28 (1789506568) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0381488 s, 537 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0628045 s, 326 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (8s) == sanityn test 122: directory size is consistent across mounts ========================================================== 17:09:37 (1789506577) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 17:09:46 (1789506586) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1266 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1324 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1344 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1087 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.122 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1605 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1544 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2197 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3951 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3409 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3534 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3217 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3699 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.387 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3507 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4803 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3728 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4555 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9db3c736a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9db3c736a800.ost_server_uuid in FULL state after 0 sec PASS 200 (80s) cleanup: ====================================================== == sanityn test complete, duration 9406 sec ============== 17:11:08 (1789506668) === sanityn: start cleanup 17:11:09 (1789506669) === Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg329-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 17:14:59 (1789506899) ===