-----============= acceptance-small: sanityn ============----- Tue Sep 29 00:52:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg240-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 00:53:05 (1790657585) === Starting client oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Mount client oleg240-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Started clients oleg240-client.virtnet: 192.168.202.140@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg240-client.virtnet: executing check_config_client /mnt/lustre oleg240-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg240-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff919049be5000.idle_timeout=debug osc.lustre-OST0000-osc-ffff919061f51800.idle_timeout=debug osc.lustre-OST0001-osc-ffff919049be5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff919061f51800.idle_timeout=debug disable quota as required oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 00:53:50 (1790657630) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527921 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 00:53:53 (1790657633) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 00:54:03 (1790657643) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 00:54:12 (1790657652) -rw-r--r-- 1 root root 0 Sep 29 00:54 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 00:54:20 (1790657660) -rw-r--r-- 1 root root 0 Sep 29 00:54 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 00:54:28 (1790657668) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 00:54:36 (1790657676) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 00:54:45 (1790657685) total 1 -rwxrwxrwx 1 root root 0 Sep 29 00:54 f2a -rwxrwxrwx 1 root root 0 Sep 29 00:54 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 29 00:54 f2a -rwxrwxrwx 1 root root 0 Sep 29 00:54 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00835394 s, 61.3 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 00:54:54 (1790657694) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 00:54:58 (1790657698) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299686 s, 7.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 00:55:07 (1790657707) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 00:55:14 (1790657714) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 00:55:22 (1790657722) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 00:55:29 (1790657729) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 00:55:37 (1790657737) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 00:55:44 (1790657744) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 00:55:51 (1790657751) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 00:55:59 (1790657759) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659798 s, 15.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0391868 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 00:56:08 (1790657768) 1+0 records in 1+0 records out 1 byte copied, 0.0192548 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00674326 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0093583 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00895634 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00943917 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0230691 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144955 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0163612 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00870589 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0391353 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0164236 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0141454 s, 0.1 kB/s PASS 10a (11s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 00:56:19 (1790657779) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0201656 s, 152 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0805602 s, 50.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000306296 s, 10.0 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 00:56:27 (1790657787) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 00:56:36 (1790657796) warning: '-runtime' deprecated, use '-t runtime' instead - link 273 (time 1790657809.16 total 10.00 last 27.29) - link 539 (time 1790657819.17 total 20.02 last 26.56) - link 737 (time 1790657829.17 total 30.02 last 19.80) - link 997 (time 1790657839.18 total 40.03 last 25.98) - link 1345 (time 1790657849.21 total 50.05 last 34.71) - link 1636 (time 1790657859.22 total 60.07 last 29.05) - link 1934 (time 1790657869.25 total 70.09 last 29.73) - link 2222 (time 1790657879.26 total 80.10 last 28.77) - link 2527 (time 1790657889.28 total 90.13 last 30.42) total: 2830 link in 100.01 seconds: 28.30 ops/second using seed 531501279 running for 100 seconds - stat 5162 (time 1790657811 ; total 11 ; last 11) - stat 9808 (time 1790657822 ; total 22 ; last 11) - stat 10000 (time 1790657822 ; total 22 ; last 0) - stat 13463 (time 1790657833 ; total 33 ; last 11) - stat 19515 (time 1790657844 ; total 44 ; last 11) - stat 20000 (time 1790657844 ; total 44 ; last 0) - stat 24887 (time 1790657855 ; total 55 ; last 11) - stat 27931 (time 1790657866 ; total 66 ; last 11) - stat 30000 (time 1790657873 ; total 73 ; last 7) - stat 33020 (time 1790657884 ; total 84 ; last 11) - stat 36375 (time 1790657895 ; total 95 ; last 11) total: 37953 stats in 100 seconds: 379.529999 stats/second - unlinked 0 (time 1790657850 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2830) error: No such file or directory total: 2829 unlinks in 80 seconds: 35.362499 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21454) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21456) - No such process PASS 12 (140s) == sanityn test 13: test directory page revocation ======= 00:58:56 (1790657936) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 00:59:05 (1790657945) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 22639 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 00:59:13 (1790657953) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 23218 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 00:59:22 (1790657962) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 23800 Terminated $DIR1/$tdir/sleep 60 PASS 14b (9s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 00:59:30 (1790657970) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 24382 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 00:59:38 (1790657978) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 24964 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 00:59:47 (1790657987) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:00:43 (1790658043) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.425014 s, 19.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.366548 s, 22.9 MB/s Chance of close/open is 1 in 50 Seed set to 8048 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000003[0] 1790658049.348602 write 0x533335 thru 0x533c86 (0x952 bytes) 000004[0] 1790658049.374227 punch from 0x156d2d to 0x161739 (0xaa0d bytes) 000005[1] 1790658049.385232 read 0x520c90 thru 0x522ae1 (0x1e52 bytes) 000006[0] 1790658049.399957 mapread 0x1efd75 thru 0x1f7a6d (0x7cf9 bytes) 000007[1] 1790658049.429106 punch from 0x4a863e to 0x4acebe (0x4881 bytes) 000010[0] 1790658049.584228 mapwrite 0x707825 thru 0x713857 (0xc033 bytes) 000013[1] 1790658049.836230 mapwrite 0x8cf8e7 thru 0x8dcff8 (0xd712 bytes) 000016[0] 1790658050.012338 punch from 0x743e32 to 0x750722 (0xc8f1 bytes) 000018[1] 1790658050.054575 write 0x263c0f thru 0x2728c5 (0xecb7 bytes) truncating to largest ever: 0x75698b 000020[1] 1790658050.091627 trunc from 0x8dcff9 to 0x75698a (0x18666e bytes) 000021[0] 1790658050.151538 mapwrite 0x25cc3f thru 0x25d4f8 (0x8ba bytes) 000024[1] 1790658050.293480 trunc from 0x75698b to 0x16adf7 (0x5ebb93 bytes) 000025[0] 1790658050.361256 mapread 0x060680 thru 0x06842f (0x7db0 bytes) 000026[0] 1790658050.413601 read 0x127cc0 thru 0x131d83 (0xa0c4 bytes) 000027[0] 1790658050.423738 mapread 0x0ec18d thru 0x0f50c7 (0x8f3b bytes) 000028[1] 1790658050.449452 punch from 0x0d0a2a to 0x0d0ccb (0x2a2 bytes) 000029[0] 1790658050.476568 mapread 0x015553 thru 0x01ee72 (0x9920 bytes) 000031[1] 1790658050.606837 write 0x1d035f thru 0x1dd377 (0xd019 bytes) 000032[1] 1790658050.623105 write 0x78db2c thru 0x79366f (0x5b44 bytes) 000034[1] 1790658050.648566 mapread 0x49d4d3 thru 0x4ab152 (0xdc80 bytes) 000035[0] 1790658050.671993 punch from 0x1911cf to 0x1a0563 (0xf395 bytes) 000037[0] 1790658050.730551 mapread 0x31abbb thru 0x325217 (0xa65d bytes) 000039[0] 1790658050.881106 trunc from 0x793670 to 0x3ee1be (0x3a54b1 bytes) 000041[1] 1790658050.953567 write 0x96ab70 thru 0x976f16 (0xc3a7 bytes) 000042[1] 1790658050.996679 read 0x28e2b4 thru 0x29151d (0x326a bytes) 000044[1] 1790658051.037503 read 0x0bb74a thru 0x0bede4 (0x369b bytes) 000045[1] 1790658051.062316 punch from 0x819f64 to 0x826aa7 (0xcb44 bytes) 000048[0] 1790658051.117343 mapread 0x265b51 thru 0x268ecb (0x337b bytes) 000049[1] 1790658051.176645 mapread 0x6eb405 thru 0x6f1305 (0x5f01 bytes) truncating to largest ever: 0x7b1fbf 000052[0] 1790658051.198554 trunc from 0x976f17 to 0x7b1fbe (0x1c4f58 bytes) 000054[1] 1790658051.296239 write 0x394aaf thru 0x3a259c (0xdaee bytes) 000055[1] 1790658051.350460 read 0x2416cb thru 0x250888 (0xf1be bytes) 000057[1] 1790658051.393702 mapread 0x4129ec thru 0x41e463 (0xba78 bytes) 000058[0] 1790658051.471726 mapwrite 0x48aa50 thru 0x49542c (0xa9dd bytes) 000061[1] 1790658051.789314 mapwrite 0x4001c7 thru 0x407f2f (0x7d69 bytes) 000064[0] 1790658051.946181 mapread 0x4b41a2 thru 0x4c30a5 (0xef04 bytes) 000069[1] 1790658052.000248 mapread 0x5bab6f thru 0x5bf9fc (0x4e8e bytes) 000071[0] 1790658052.070217 write 0x33b78c thru 0x33c0b6 (0x92b bytes) 000072[0] 1790658052.137804 read 0x411dbc thru 0x41e58d (0xc7d2 bytes) 000073[0] 1790658052.159144 write 0x9e08bd thru 0x9ed12c (0xc870 bytes) 000074[1] 1790658052.174222 mapread 0x472478 thru 0x474e6b (0x29f4 bytes) 000077[1] 1790658052.210832 mapwrite 0x41bd56 thru 0x4225d2 (0x687d bytes) truncating to largest ever: 0x88aff1 000078[1] 1790658052.284878 trunc from 0x9ed12d to 0x88aff0 (0x16213c bytes) 000083[1] 1790658052.375644 write 0x4fe38f thru 0x503cfe (0x5970 bytes) 000084[0] 1790658052.406219 mapwrite 0x7ec56c thru 0x7f2307 (0x5d9c bytes) 000085[0] 1790658052.539533 mapread 0x0d872c thru 0x0e8445 (0xfd1a bytes) 000086[1] 1790658052.592436 punch from 0x3ad5bc to 0x3b288f (0x52d4 bytes) 000090[1] 1790658052.663436 punch from 0x059378 to 0x05a6d7 (0x1360 bytes) 000091[1] 1790658052.679966 mapwrite 0x25a10b thru 0x25c2b0 (0x21a6 bytes) 000094[1] 1790658052.837438 mapread 0x45e137 thru 0x4675a3 (0x946d bytes) 000095[0] 1790658052.869306 write 0x837d49 thru 0x843efc (0xc1b4 bytes) 000096[0] 1790658052.896709 trunc from 0x88aff1 to 0x2fc0ac (0x58ef44 bytes) 000097[1] 1790658052.926575 mapwrite 0x6d7ec5 thru 0x6e3dd9 (0xbf15 bytes) 000099[1] 1790658053.101171 read 0x603c38 thru 0x609659 (0x5a22 bytes) 000100[1] 1790658053.121390 trunc from 0x6e3dda to 0x63ce80 (0xa6f59 bytes) 000102[1] 1790658053.207553 write 0x41611a thru 0x419bf2 (0x3ad9 bytes) 000103[0] 1790658053.261181 punch from 0x36227d to 0x365eb5 (0x3c39 bytes) 000104[0] 1790658053.336059 read 0x57b540 thru 0x586c69 (0xb72a bytes) 000105[0] 1790658053.389741 mapwrite 0x671736 thru 0x677533 (0x5dfe bytes) 000106[0] 1790658053.552038 mapwrite 0x9c1dd9 thru 0x9c92f5 (0x751d bytes) 000109[0] 1790658053.806637 mapread 0x888de1 thru 0x8916fe (0x891e bytes) 000110[0] 1790658053.838416 read 0x71c9e0 thru 0x7200e3 (0x3704 bytes) 000113[1] 1790658053.963454 read 0x0b4555 thru 0x0b6a29 (0x24d5 bytes) 000114[1] 1790658054.039515 read 0x6084e9 thru 0x60c298 (0x3db0 bytes) 000116[1] 1790658054.112866 read 0x3a4458 thru 0x3a5934 (0x14dd bytes) 000117[1] 1790658054.143442 trunc from 0x9c92f6 to 0x146e41 (0x8824b4 bytes) 000118[1] 1790658054.238179 trunc from 0x146e42 to 0x67e927 (0x537ae6 bytes) 000119[1] 1790658054.266640 mapwrite 0x346122 thru 0x348f4c (0x2e2b bytes) 000120[0] 1790658054.347201 trunc from 0x67e928 to 0x2c4db3 (0x3b9b74 bytes) 000121[1] 1790658054.432115 read 0x22b372 thru 0x2303bd (0x504c bytes) 000122[1] 1790658054.540745 punch from 0x03245e to 0x03e362 (0xbf05 bytes) 000129[1] 1790658054.766611 read 0x104e09 thru 0x10f2dd (0xa4d5 bytes) 000132[1] 1790658054.905142 mapread 0x01c0d6 thru 0x02ae08 (0xed33 bytes) 000133[1] 1790658054.932640 punch from 0x0f5ac0 to 0x1053e6 (0xf927 bytes) 000134[0] 1790658054.956876 trunc from 0x2c4db4 to 0x40473f (0x13f98c bytes) 000135[0] 1790658055.023159 trunc from 0x404740 to 0x2d4db7 (0x12f988 bytes) 000137[1] 1790658055.100201 trunc from 0x2d4db8 to 0x0f6077 (0x1ded40 bytes) 000138[1] 1790658055.152102 mapwrite 0x037055 thru 0x03d978 (0x6924 bytes) 000139[0] 1790658055.251343 mapwrite 0x925c0c thru 0x92cd3c (0x7131 bytes) 000141[0] 1790658055.479217 punch from 0x454c16 to 0x4559aa (0xd95 bytes) truncating to largest ever: 0x8adf7c 000143[1] 1790658055.502852 trunc from 0x92cd3d to 0x8adf7b (0x7edc1 bytes) 000145[0] 1790658055.611686 mapwrite 0x15db02 thru 0x1602d5 (0x27d4 bytes) 000146[0] 1790658055.807361 mapwrite 0x985de6 thru 0x9860b3 (0x2ce bytes) 000147[1] 1790658056.013499 punch from 0x60ddd3 to 0x6136cc (0x58fa bytes) 000148[0] 1790658056.055107 trunc from 0x9860b4 to 0x474640 (0x511a73 bytes) 000150[1] 1790658056.150342 write 0x1c87c1 thru 0x1d01cf (0x7a0f bytes) 000152[0] 1790658056.247680 write 0x2f5c2a thru 0x2fdf9e (0x8375 bytes) 000153[0] 1790658056.283703 write 0x5cdf76 thru 0x5dc9c9 (0xea54 bytes) 000154[1] 1790658056.316379 mapwrite 0x9ee5cc thru 0x9fa54e (0xbf83 bytes) 000155[0] 1790658056.590031 mapwrite 0x7654ae thru 0x7688b4 (0x3407 bytes) 000156[0] 1790658056.840519 trunc from 0x9fa54f to 0x1dc62f (0x81df1f bytes) 000157[1] 1790658056.901774 read 0x1cb481 thru 0x1d10ed (0x5c6d bytes) 000158[0] 1790658056.975570 trunc from 0x1dc630 to 0x6f2773 (0x516144 bytes) 000160[1] 1790658057.017011 punch from 0x41ef39 to 0x426e51 (0x7f19 bytes) 000161[1] 1790658057.048669 mapwrite 0x050d6c thru 0x05af9d (0xa232 bytes) 000163[0] 1790658057.166127 read 0x4b8be2 thru 0x4c5124 (0xc543 bytes) 000166[0] 1790658057.227350 write 0x0da945 thru 0x0e1570 (0x6c2c bytes) truncating to largest ever: 0x9d01e0 000168[0] 1790658057.244855 trunc from 0x6f2774 to 0x9d01df (0x2dda6c bytes) 000169[1] 1790658057.313543 mapwrite 0x3adad5 thru 0x3adda0 (0x2cc bytes) 000171[1] 1790658057.523654 write 0x5d377a thru 0x5d6ca0 (0x3527 bytes) 000172[1] 1790658057.580556 mapwrite 0x4ef23d thru 0x4f570c (0x64d0 bytes) 000173[1] 1790658057.686930 mapwrite 0x850a76 thru 0x855705 (0x4c90 bytes) 000175[0] 1790658057.795138 write 0x8e5b73 thru 0x8ee875 (0x8d03 bytes) 000176[1] 1790658057.891328 mapwrite 0x6bee20 thru 0x6c049f (0x1680 bytes) 000177[1] 1790658058.081206 punch from 0x250c12 to 0x259d58 (0x9147 bytes) 000178[0] 1790658058.136275 read 0x7da1a1 thru 0x7dac0c (0xa6c bytes) 000179[1] 1790658058.230090 mapwrite 0x361b1c thru 0x36e62d (0xcb12 bytes) 000180[1] 1790658058.384074 read 0x4bd52a thru 0x4c6635 (0x910c bytes) 000181[0] 1790658058.422341 punch from 0x581e52 to 0x588326 (0x64d5 bytes) 000184[1] 1790658058.571385 write 0x0a48d2 thru 0x0a6af7 (0x2226 bytes) 000186[0] 1790658058.652681 punch from 0x85fbba to 0x862799 (0x2be0 bytes) 000189[1] 1790658058.720410 read 0x14ff3b thru 0x15ae9c (0xaf62 bytes) 000190[1] 1790658058.752111 write 0x4de36a thru 0x4e7863 (0x94fa bytes) 000192[1] 1790658058.792158 punch from 0x497107 to 0x49b2d6 (0x41d0 bytes) 000193[1] 1790658058.820170 mapread 0x5a4bb3 thru 0x5a78de (0x2d2c bytes) 000194[0] 1790658058.831470 mapwrite 0x733220 thru 0x73ced2 (0x9cb3 bytes) 000195[0] 1790658058.912108 punch from 0x1fc7e4 to 0x2095c7 (0xcde4 bytes) 000198[1] 1790658059.006300 read 0x03752d thru 0x03b018 (0x3aec bytes) 000199[0] 1790658059.030394 trunc from 0x9d01e0 to 0x7f505c (0x1db183 bytes) 000200[1] 1790658059.073380 punch from 0x46b121 to 0x47a61f (0xf4ff bytes) 000201[0] 1790658059.103286 mapread 0x5073c1 thru 0x50a80a (0x344a bytes) 000202[1] 1790658059.195348 mapwrite 0x001db8 thru 0x00ac51 (0x8e9a bytes) 000203[1] 1790658059.347210 trunc from 0x7f505d to 0x76ef28 (0x86134 bytes) 000207[0] 1790658059.529665 trunc from 0x76ef29 to 0x357835 (0x4176f3 bytes) 000208[1] 1790658059.592037 read 0x279428 thru 0x27ce4a (0x3a23 bytes) 000209[1] 1790658059.615809 write 0x8f3b00 thru 0x8f9246 (0x5747 bytes) 000210[1] 1790658059.656631 read 0x79a3fc thru 0x7a9b62 (0xf767 bytes) 000211[0] 1790658059.698303 write 0x8face4 thru 0x906d16 (0xc033 bytes) 000212[0] 1790658059.760535 read 0x0562b2 thru 0x063126 (0xce75 bytes) 000214[1] 1790658059.771114 punch from 0x17f5f6 to 0x1876e0 (0x80eb bytes) 000217[1] 1790658059.856072 trunc from 0x906d17 to 0x8bd20c (0x49b0a bytes) 000219[1] 1790658059.955403 read 0x6be73b thru 0x6c1cae (0x3574 bytes) 000221[1] 1790658060.015672 trunc from 0x8bd20d to 0x065d8f (0x85747d bytes) 000222[1] 1790658060.077386 mapread 0x01c02d thru 0x029494 (0xd468 bytes) 000223[1] 1790658060.150371 read 0x00035c thru 0x006496 (0x613b bytes) 000224[0] 1790658060.184567 write 0x64ce4e thru 0x64f10b (0x22be bytes) 000225[0] 1790658060.221575 mapread 0x08f650 thru 0x0990b1 (0x9a62 bytes) 000227[0] 1790658060.321024 mapwrite 0x11e63b thru 0x127ce5 (0x96ab bytes) 000228[1] 1790658060.440011 punch from 0x1c4960 to 0x1cb0cb (0x676c bytes) 000229[0] 1790658060.471641 write 0x3ee2ba thru 0x3f20e8 (0x3e2f bytes) 000232[1] 1790658060.522020 punch from 0x4a0d44 to 0x4adb16 (0xcdd3 bytes) 000234[0] 1790658060.608147 mapwrite 0x0bcc77 thru 0x0bf1ea (0x2574 bytes) 000235[0] 1790658060.714523 mapwrite 0x6471b9 thru 0x64f4fb (0x8343 bytes) 000236[0] 1790658060.858033 punch from 0x52bb9c to 0x538376 (0xc7db bytes) 000237[1] 1790658060.867655 mapread 0x1e827f thru 0x1f5838 (0xd5ba bytes) 000238[1] 1790658061.025984 trunc from 0x64f4fc to 0x665960 (0x16465 bytes) 000239[1] 1790658061.197174 read 0x55c619 thru 0x56328c (0x6c74 bytes) 000240[1] 1790658061.267345 mapwrite 0x3e7f1e thru 0x3e9fc8 (0x20ab bytes) 000241[0] 1790658061.406089 read 0x35d999 thru 0x36d82a (0xfe92 bytes) 000242[1] 1790658061.457602 mapwrite 0x0d6eb3 thru 0x0e4908 (0xda56 bytes) 000243[1] 1790658061.827539 mapwrite 0x9fa3a4 thru 0x9fb82a (0x1487 bytes) 000244[1] 1790658061.946859 write 0x2da895 thru 0x2def7f (0x46eb bytes) 000246[1] 1790658062.025213 trunc from 0x9fb82b to 0x704d53 (0x2f6ad7 bytes) 000247[0] 1790658062.089315 read 0x3a388e thru 0x3b1b8d (0xe300 bytes) 000248[1] 1790658062.163482 mapwrite 0x51055e thru 0x514339 (0x3ddc bytes) 000249[0] 1790658062.344685 mapread 0x01e17b thru 0x01fa1d (0x18a3 bytes) 000250[1] 1790658062.416540 mapwrite 0x6f5bab thru 0x702b98 (0xcfee bytes) 000252[1] 1790658062.663031 punch from 0x45282d to 0x45e16c (0xb940 bytes) 000254[0] 1790658062.718903 punch from 0x54d29b to 0x5545b8 (0x731e bytes) 000256[1] 1790658062.776688 read 0x53b83c thru 0x542bcb (0x7390 bytes) 000258[0] 1790658062.837157 read 0x6e2092 thru 0x6edcf0 (0xbc5f bytes) 000260[1] 1790658062.848694 trunc from 0x704d54 to 0x55e92a (0x1a6429 bytes) 000261[1] 1790658062.926514 punch from 0x37cb28 to 0x38025c (0x3735 bytes) 000263[0] 1790658062.952017 read 0x3fd110 thru 0x40061d (0x350e bytes) 000264[1] 1790658062.989664 trunc from 0x55e92b to 0x8f425b (0x395931 bytes) 000265[1] 1790658063.036062 mapread 0x1eaa66 thru 0x1f26cb (0x7c66 bytes) 000266[1] 1790658063.083000 write 0x20f8dd thru 0x213c9d (0x43c1 bytes) 000269[1] 1790658063.095501 read 0x6b7f16 thru 0x6c4c14 (0xccff bytes) 000270[0] 1790658063.123229 trunc from 0x8f425c to 0x071c72 (0x8825e9 bytes) 000272[0] 1790658063.190452 mapread 0x051da5 thru 0x05ee46 (0xd0a2 bytes) 000273[0] 1790658063.226477 punch from 0x03098c to 0x0346c4 (0x3d39 bytes) 000276[1] 1790658063.334133 read 0x040adf thru 0x04124b (0x76d bytes) 000277[1] 1790658063.400492 punch from 0x032719 to 0x0346b0 (0x1f98 bytes) 000278[1] 1790658063.462962 trunc from 0x071c73 to 0x1acd5a (0x13b0e8 bytes) 000279[0] 1790658063.566656 read 0x0ab795 thru 0x0b0123 (0x498f bytes) 000280[0] 1790658063.640547 trunc from 0x1acd5b to 0x5be83a (0x411ae0 bytes) 000281[0] 1790658063.692689 read 0x0a695a thru 0x0b2825 (0xbecc bytes) 000282[1] 1790658063.726652 mapwrite 0x7ec876 thru 0x7f1b32 (0x52bd bytes) 000283[1] 1790658064.014794 mapread 0x1a918a thru 0x1b8e83 (0xfcfa bytes) 000284[1] 1790658064.038323 trunc from 0x7f1b33 to 0x194679 (0x65d4b9 bytes) 000285[0] 1790658064.101645 mapread 0x145b5d thru 0x14e636 (0x8ada bytes) 000286[1] 1790658064.157643 punch from 0x042867 to 0x0509d3 (0xe16d bytes) 000288[0] 1790658064.174375 mapwrite 0x8e57c9 thru 0x8f163a (0xbe72 bytes) 000290[1] 1790658064.312094 mapwrite 0x7878fe thru 0x795a40 (0xe143 bytes) 000291[0] 1790658064.479205 trunc from 0x8f163b to 0x5de6d3 (0x312f67 bytes) 000293[1] 1790658064.583257 punch from 0x5761bf to 0x576a0d (0x84f bytes) 000294[0] 1790658064.621679 mapread 0x5ce68f thru 0x5dd85a (0xf1cc bytes) 000295[0] 1790658064.692048 punch from 0x1c80e3 to 0x1d01b7 (0x80d5 bytes) 000297[1] 1790658064.699151 mapread 0x137f48 thru 0x138507 (0x5c0 bytes) 000301[1] 1790658064.754594 mapwrite 0x65d32b thru 0x663bc9 (0x689f bytes) 000303[0] 1790658064.937536 trunc from 0x663bca to 0x74e3e4 (0xea81b bytes) 000304[1] 1790658065.036383 read 0x005a86 thru 0x012d4a (0xd2c5 bytes) 000306[1] 1790658065.118194 mapread 0x56a7ea thru 0x57163d (0x6e54 bytes) 000308[1] 1790658065.188622 mapwrite 0x003ea1 thru 0x0072f7 (0x3457 bytes) 000309[0] 1790658065.394231 mapwrite 0x593c11 thru 0x59c087 (0x8477 bytes) 000310[1] 1790658065.513509 trunc from 0x74e3e5 to 0x092438 (0x6bbfac bytes) 000311[1] 1790658065.583657 punch from 0x07e2ed to 0x085489 (0x719d bytes) 000314[1] 1790658065.669565 write 0x441ec4 thru 0x44d7a9 (0xb8e6 bytes) 000315[1] 1790658065.680595 mapread 0x0f6cd9 thru 0x102604 (0xb92c bytes) 000318[1] 1790658065.768962 punch from 0x427555 to 0x42c1ad (0x4c59 bytes) 000319[0] 1790658065.808026 trunc from 0x44d7aa to 0x7490cf (0x2fb926 bytes) 000322[0] 1790658065.886447 mapread 0x118904 thru 0x118d35 (0x432 bytes) 000324[0] 1790658065.957888 mapwrite 0x02a6cc thru 0x03718c (0xcac1 bytes) 000325[0] 1790658066.049047 mapwrite 0x5786c7 thru 0x57d395 (0x4ccf bytes) 000326[0] 1790658066.341323 mapwrite 0x6334d0 thru 0x63500b (0x1b3c bytes) 000327[0] 1790658066.422251 trunc from 0x7490d0 to 0x77e12a (0x3505b bytes) 000329[1] 1790658066.440975 mapread 0x6d4a5c thru 0x6e17cd (0xcd72 bytes) 000330[1] 1790658066.516804 mapwrite 0x756794 thru 0x765dbf (0xf62c bytes) 000331[1] 1790658066.619297 write 0x15c219 thru 0x15d914 (0x16fc bytes) 000332[1] 1790658066.641931 mapread 0x4ae05f thru 0x4b01a1 (0x2143 bytes) 000333[0] 1790658066.682288 read 0x2ed011 thru 0x2f3fe0 (0x6fd0 bytes) 000334[0] 1790658066.723327 mapread 0x6f5e5c thru 0x6fb73f (0x58e4 bytes) 000335[0] 1790658066.788348 read 0x711777 thru 0x7137cc (0x2056 bytes) 000338[1] 1790658066.789540 mapwrite 0x69290c thru 0x696f9e (0x4693 bytes) 000339[0] 1790658066.905138 mapwrite 0x5e1e49 thru 0x5ec4af (0xa667 bytes) 000343[1] 1790658067.047959 trunc from 0x77e12b to 0x633e2f (0x14a2fb bytes) 000345[1] 1790658067.162587 write 0x59107a thru 0x5923e3 (0x136a bytes) 000346[1] 1790658067.189949 trunc from 0x633e30 to 0x218d93 (0x41b09c bytes) 000347[0] 1790658067.237554 write 0x3486d1 thru 0x3512af (0x8bdf bytes) 000349[1] 1790658067.316092 mapwrite 0x4b07aa thru 0x4b86e6 (0x7f3d bytes) 000351[1] 1790658067.652479 punch from 0x2221cd to 0x231374 (0xf1a8 bytes) 000352[1] 1790658067.717748 write 0x947fdb thru 0x956a60 (0xea86 bytes) 000353[1] 1790658067.758255 read 0x235788 thru 0x239206 (0x3a7f bytes) 000356[1] 1790658067.828269 trunc from 0x956a61 to 0x724e37 (0x231c29 bytes) 000357[0] 1790658067.871496 trunc from 0x724e38 to 0x999e46 (0x27500f bytes) 000359[0] 1790658067.947956 trunc from 0x999e47 to 0x3a46aa (0x5f579c bytes) 000360[1] 1790658067.998232 punch from 0x2dfd73 to 0x2ee333 (0xe5c1 bytes) 000361[1] 1790658068.031254 mapread 0x0d3ed9 thru 0x0e3c94 (0xfdbc bytes) 000365[0] 1790658068.244972 punch from 0x0426e5 to 0x0440c6 (0x19e2 bytes) 000366[1] 1790658068.261808 punch from 0x0e71d6 to 0x0ef18f (0x7fba bytes) 000368[0] 1790658068.328024 mapread 0x0c3900 thru 0x0c41f7 (0x8f8 bytes) 000369[1] 1790658068.355081 mapread 0x1a04e9 thru 0x1a2a4f (0x2567 bytes) 000370[1] 1790658068.378517 trunc from 0x3a46ab to 0x724de1 (0x380737 bytes) 000372[0] 1790658068.498313 punch from 0x6cb816 to 0x6d4385 (0x8b70 bytes) 000373[1] 1790658068.574897 mapread 0x428c8b thru 0x433407 (0xa77d bytes) 000375[0] 1790658068.678963 read 0x0620f0 thru 0x06757f (0x5490 bytes) 000376[0] 1790658068.707114 mapwrite 0x1d7a08 thru 0x1e459b (0xcb94 bytes) 000377[1] 1790658068.848521 punch from 0x59963a to 0x59a547 (0xf0e bytes) 000382[1] 1790658068.947264 mapwrite 0x683f85 thru 0x69014c (0xc1c8 bytes) 000383[0] 1790658069.059812 mapwrite 0x92257e thru 0x92466a (0x20ed bytes) 000386[0] 1790658069.303839 write 0x0ee325 thru 0x0f02ea (0x1fc6 bytes) 000387[1] 1790658069.334324 trunc from 0x92466b to 0x3c90e6 (0x55b584 bytes) 000389[1] 1790658069.442362 mapwrite 0x067b30 thru 0x06b51e (0x39ef bytes) 000390[1] 1790658069.583612 punch from 0x224cfb to 0x233c63 (0xef69 bytes) 000394[0] 1790658069.595404 mapwrite 0x4deae3 thru 0x4e6382 (0x78a0 bytes) 000396[1] 1790658069.819311 read 0x1bf45d thru 0x1c1630 (0x21d4 bytes) 000399[0] 1790658069.865022 trunc from 0x4e6383 to 0x28b564 (0x25ae1e bytes) 000401[0] 1790658069.908967 read 0x25c5cd thru 0x26682a (0xa25e bytes) 000402[1] 1790658070.000347 mapread 0x0dcbc6 thru 0x0e44ea (0x7925 bytes) 000403[1] 1790658070.043796 mapread 0x01137f thru 0x016f3c (0x5bbe bytes) 000404[1] 1790658070.070724 trunc from 0x28b565 to 0x4fe81b (0x2732b7 bytes) 000407[1] 1790658070.164818 write 0x39e1c1 thru 0x3a843f (0xa27f bytes) 000409[0] 1790658070.198003 mapread 0x4e88f9 thru 0x4ec447 (0x3b4f bytes) 000410[0] 1790658070.252872 mapread 0x386811 thru 0x392a00 (0xc1f0 bytes) 000411[0] 1790658070.319720 read 0x2451c1 thru 0x248fa6 (0x3de6 bytes) 000412[1] 1790658070.324837 trunc from 0x4fe81c to 0x645aa4 (0x147289 bytes) 000413[1] 1790658070.380540 mapread 0x156457 thru 0x1657b6 (0xf360 bytes) 000415[1] 1790658070.455796 write 0x83c3c0 thru 0x844420 (0x8061 bytes) 000416[1] 1790658070.457517 mapwrite 0x823529 thru 0x82cbad (0x9685 bytes) 000417[0] 1790658070.631393 mapread 0x1011f6 thru 0x1026f7 (0x1502 bytes) 000418[1] 1790658070.692843 write 0x78a8aa thru 0x790b7f (0x62d6 bytes) 000419[1] 1790658070.703032 mapread 0x1e6dc1 thru 0x1e94b2 (0x26f2 bytes) 000423[0] 1790658070.803660 mapwrite 0x5e7b7b thru 0x5ea5ef (0x2a75 bytes) 000424[1] 1790658070.929361 trunc from 0x844421 to 0x40d350 (0x4370d0 bytes) 000425[1] 1790658070.990450 read 0x104901 thru 0x10c1f4 (0x78f4 bytes) 000426[1] 1790658071.000234 punch from 0x21f295 to 0x222a49 (0x37b5 bytes) 000429[1] 1790658071.027279 mapwrite 0x25762c thru 0x25baa2 (0x4477 bytes) 000430[0] 1790658071.154432 write 0x381076 thru 0x38bfd9 (0xaf64 bytes) 000431[1] 1790658071.241606 mapwrite 0x1bdbbd thru 0x1bdcd1 (0x115 bytes) 000432[0] 1790658071.370555 mapwrite 0x7ca8ec thru 0x7ce162 (0x3877 bytes) 000433[1] 1790658071.486587 read 0x611098 thru 0x6166c9 (0x5632 bytes) 000434[1] 1790658071.517532 trunc from 0x7ce163 to 0x569854 (0x26490e bytes) 000435[1] 1790658071.573981 mapwrite 0x3c61ec thru 0x3c6509 (0x31e bytes) 000436[0] 1790658071.628036 mapread 0x3332ff thru 0x33a2db (0x6fdd bytes) 000438[0] 1790658071.696097 trunc from 0x569855 to 0x0f9f6e (0x46f8e6 bytes) 000440[1] 1790658071.735915 read 0x0a1cfa thru 0x0a7be0 (0x5ee7 bytes) 000442[0] 1790658071.756616 mapwrite 0x092d31 thru 0x0a1756 (0xea26 bytes) 000443[1] 1790658071.897585 trunc from 0x0f9f6f to 0x4d1f65 (0x3d7ff7 bytes) 000444[0] 1790658071.939338 read 0x289c01 thru 0x28ac25 (0x1025 bytes) 000445[1] 1790658072.084314 mapread 0x38febc thru 0x38ffcd (0x112 bytes) 000446[0] 1790658072.115372 read 0x0ff9ef thru 0x10b1ed (0xb7ff bytes) 000447[0] 1790658072.136440 trunc from 0x4d1f66 to 0x9cfd78 (0x4fde13 bytes) 000448[1] 1790658072.207597 write 0x76fcdc thru 0x77eacc (0xedf1 bytes) 000449[1] 1790658072.250048 mapread 0x0b3484 thru 0x0c2566 (0xf0e3 bytes) 000450[0] 1790658072.297535 punch from 0x94e8e2 to 0x95ada3 (0xc4c2 bytes) 000451[0] 1790658072.328798 write 0x9173a1 thru 0x924171 (0xcdd1 bytes) 000452[0] 1790658072.359508 write 0x3e22c5 thru 0x3ef490 (0xd1cc bytes) 000453[0] 1790658072.389217 write 0x6b1baf thru 0x6bec48 (0xd09a bytes) 000454[1] 1790658072.425248 read 0x9cac0c thru 0x9cfd78 (0x516d bytes) 000455[1] 1790658072.465991 write 0x0339a5 thru 0x0377b9 (0x3e15 bytes) 000456[1] 1790658072.501510 punch from 0x3b3aeb to 0x3b50e6 (0x15fc bytes) 000458[1] 1790658072.533385 read 0x3a2f3b thru 0x3aacf9 (0x7dbf bytes) 000459[0] 1790658072.562790 write 0x8f9b3b thru 0x9084f1 (0xe9b7 bytes) 000460[0] 1790658072.620550 punch from 0x13f20c to 0x1428d2 (0x36c7 bytes) 000461[1] 1790658072.651288 trunc from 0x9cfd79 to 0x7acae8 (0x223290 bytes) 000462[0] 1790658072.746000 punch from 0x5e66fa to 0x5ece1f (0x6726 bytes) 000463[1] 1790658072.776498 trunc from 0x7acae9 to 0x11a70f (0x6923d9 bytes) 000465[1] 1790658072.843733 mapread 0x0d604a thru 0x0e1768 (0xb71f bytes) 000468[1] 1790658072.900926 punch from 0x0ac8cc to 0x0ae176 (0x18ab bytes) 000469[0] 1790658072.923595 read 0x07da6e thru 0x088b87 (0xb11a bytes) 000470[0] 1790658072.969541 trunc from 0x11a710 to 0x8550d3 (0x73a9c4 bytes) 000471[0] 1790658073.049687 write 0x4f91f4 thru 0x504bf0 (0xb9fd bytes) 000477[1] 1790658073.231694 write 0x24ed16 thru 0x251605 (0x28f0 bytes) 000478[0] 1790658073.271268 write 0x831a6b thru 0x841282 (0xf818 bytes) 000479[0] 1790658073.378542 write 0x01a72c thru 0x01c860 (0x2135 bytes) 000481[0] 1790658073.387998 write 0x039290 thru 0x039cfc (0xa6d bytes) 000482[1] 1790658073.392962 punch from 0x16fe67 to 0x177a72 (0x7c0c bytes) 000486[1] 1790658073.467319 mapread 0x58b67c thru 0x593af6 (0x847b bytes) 000487[1] 1790658073.536295 mapread 0x211071 thru 0x21d69c (0xc62c bytes) 000489[0] 1790658073.597048 read 0x51f163 thru 0x52049e (0x133c bytes) 000490[0] 1790658073.602596 mapread 0x13fc9e thru 0x14034f (0x6b2 bytes) 000491[0] 1790658073.636943 write 0x5d9345 thru 0x5dc108 (0x2dc4 bytes) 000493[0] 1790658073.675534 mapread 0x727b2f thru 0x72bc1b (0x40ed bytes) 000494[0] 1790658073.698028 mapread 0x708d88 thru 0x70ad10 (0x1f89 bytes) 000495[1] 1790658073.709343 trunc from 0x8550d4 to 0x61dbad (0x237526 bytes) 000496[1] 1790658073.786521 write 0x561b3d thru 0x568809 (0x6ccd bytes) 000497[1] 1790658073.825384 mapread 0x35d16c thru 0x36817e (0xb013 bytes) 000498[1] 1790658073.888920 trunc from 0x61dbae to 0x0f0fcb (0x52cbe2 bytes) 000499[1] 1790658073.910430 write 0x0c69bb thru 0x0cc1e3 (0x5829 bytes) 000500[0] 1790658073.942597 read 0x018b70 thru 0x01acac (0x213d bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8074 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x4b1f93 000002[0] 1790658074.624939 trunc from 00000000 to 0x4b1f92 (0x4b1f93 bytes) truncating to largest ever: 0x7cfa4e 000004[1] 1790658074.668192 trunc from 0x4b1f93 to 0x7cfa4d (0x31dabb bytes) 000005[0] 1790658074.737928 trunc from 0x7cfa4e to 0x57022c (0x25f821 bytes) 000006[0] 1790658074.816190 trunc from 0x57022d to 0x00a8d9 (0x565953 bytes) 000009[1] 1790658074.838717 trunc from 0x00a8da to 0x3ad7c7 (0x3a2eee bytes) 000011[0] 1790658074.922903 mapwrite 0x77b000 thru 0x781410 (0x6411 bytes) 000012[1] 1790658075.044479 trunc from 0x781411 to 0x0aff66 (0x6d14aa bytes) 000013[0] 1790658075.097180 mapwrite 0x437000 thru 0x43df1e (0x6f1f bytes) 000014[0] 1790658075.211752 write 0x24b000 thru 0x2526a3 (0x76a4 bytes) 000015[1] 1790658075.219593 punch from 0x20ad6d to 0x21687d (0xbb11 bytes) 000018[0] 1790658075.249066 read 0x08d000 thru 0x099229 (0xc22a bytes) 000019[1] 1790658075.267258 read_OD 0x2fa000 thru 0x307fff (0xe000 bytes) 000020[1] 1790658075.285800 punch from 0x1b7a7b to 0x1c57d0 (0xdd56 bytes) 000021[0] 1790658075.301519 punch from 0x126492 to 0x135e07 (0xf976 bytes) 000022[0] 1790658075.324810 mapread 0x21b000 thru 0x21dffb (0x2ffc bytes) 000024[0] 1790658075.355871 trunc from 0x43df1f to 0x2b7e92 (0x18608c bytes) 000025[0] 1790658075.380054 mapread 0x079000 thru 0x0877d8 (0xe7d9 bytes) 000026[1] 1790658075.419510 write_OD 0x24e000 thru 0x251fff (0x4000 bytes) 000027[0] 1790658075.471941 write 0x92d000 thru 0x939a74 (0xca75 bytes) 000028[0] 1790658075.489640 read 0x467000 thru 0x475991 (0xe992 bytes) 000029[1] 1790658075.527227 mapread 0x23f000 thru 0x2446ef (0x56f0 bytes) 000032[1] 1790658075.610944 trunc from 0x939a75 to 0x02ef86 (0x90aaee bytes) 000035[1] 1790658075.670134 mapwrite 0x131000 thru 0x13a11e (0x911f bytes) 000036[1] 1790658075.768480 write_OD 0x6f9000 thru 0x6fafff (0x2000 bytes) 000038[1] 1790658075.862268 trunc from 0x6fb000 to 0x0552d8 (0x6a5d27 bytes) 000040[0] 1790658075.901397 read 0x022000 thru 0x02b43a (0x943b bytes) 000042[0] 1790658075.927684 punch from 0x038fb4 to 0x03c69d (0x36ea bytes) 000044[0] 1790658075.971458 mapread 0x007000 thru 0x013610 (0xc611 bytes) 000045[1] 1790658075.984368 punch from 0x021322 to 0x02e433 (0xd112 bytes) 000048[0] 1790658076.014608 punch from 0x019cf0 to 0x020d7d (0x708e bytes) truncating to largest ever: 0x8ae171 000050[0] 1790658076.064038 trunc from 0x0552d9 to 0x8ae170 (0x858e98 bytes) 000051[1] 1790658076.108877 punch from 0x58dbb4 to 0x598315 (0xa762 bytes) 000052[0] 1790658076.125057 punch from 0x503841 to 0x50ea3e (0xb1fe bytes) 000053[0] 1790658076.152702 trunc from 0x8ae171 to 0x1190c4 (0x7950ac bytes) 000054[0] 1790658076.174556 trunc from 0x1190c5 to 0x369ffe (0x250f3a bytes) 000057[0] 1790658076.259388 write 0x3f8000 thru 0x3fa014 (0x2015 bytes) 000058[0] 1790658076.263676 mapread 0x0b9000 thru 0x0c0215 (0x7216 bytes) 000059[1] 1790658076.303456 trunc from 0x3fa015 to 0x54a0fb (0x1500e7 bytes) 000060[1] 1790658076.372356 write_OD 0x7a4000 thru 0x7b2fff (0xf000 bytes) 000063[1] 1790658076.450433 trunc from 0x7b3000 to 0x1ad4d2 (0x605b2d bytes) 000066[0] 1790658076.570821 write 0x36c000 thru 0x371d55 (0x5d56 bytes) 000067[0] 1790658076.600853 write 0x834000 thru 0x834685 (0x686 bytes) 000068[0] 1790658076.633868 mapwrite 0x538000 thru 0x53a53e (0x253f bytes) 000069[0] 1790658076.705878 write 0x60f000 thru 0x61c6a6 (0xd6a7 bytes) 000072[0] 1790658076.766392 read 0x00b000 thru 0x0109bf (0x59c0 bytes) 000073[1] 1790658076.775629 read_OD 0x726000 thru 0x732fff (0xd000 bytes) 000075[0] 1790658076.817495 trunc from 0x834686 to 0x2952bf (0x59f3c6 bytes) 000076[0] 1790658076.853672 read 0x087000 thru 0x08b07b (0x407c bytes) 000077[0] 1790658076.934028 trunc from 0x2952c0 to 0x89a749 (0x60548a bytes) 000078[1] 1790658076.954177 mapwrite 0x7d9000 thru 0x7e8c80 (0xfc81 bytes) 000079[0] 1790658077.060116 mapwrite 0x3ad000 thru 0x3b5049 (0x804a bytes) 000081[1] 1790658077.263857 read_OD 0x225000 thru 0x225fff (0x1000 bytes) 000082[1] 1790658077.299276 write_OD 0x1f9000 thru 0x1fdfff (0x5000 bytes) 000083[1] 1790658077.357857 punch from 0x098a5d to 0x09d591 (0x4b35 bytes) 000085[1] 1790658077.394573 trunc from 0x89a74a to 0x3c942c (0x4d131d bytes) 000086[1] 1790658077.473964 mapread 0x287000 thru 0x2878e1 (0x8e2 bytes) 000088[1] 1790658077.511018 mapwrite 0x113000 thru 0x11e3f3 (0xb3f4 bytes) 000091[0] 1790658077.635242 mapread 0x07e000 thru 0x082bea (0x4beb bytes) 000092[1] 1790658077.690128 write_OD 0x63a000 thru 0x643fff (0xa000 bytes) 000094[1] 1790658077.766473 mapread 0x39d000 thru 0x3a6f0e (0x9f0f bytes) 000095[0] 1790658077.795668 mapread 0x09c000 thru 0x09ff79 (0x3f7a bytes) 000100[1] 1790658077.815030 trunc from 0x644000 to 0x20e269 (0x435d96 bytes) 000101[1] 1790658077.850888 mapwrite 0x41e000 thru 0x4210e7 (0x30e8 bytes) 000102[1] 1790658077.954447 mapread 0x24e000 thru 0x2551fd (0x71fe bytes) 000103[1] 1790658078.011259 write_OD 0x0a5000 thru 0x0aafff (0x6000 bytes) 000104[0] 1790658078.162816 write 0x917000 thru 0x91ac84 (0x3c85 bytes) 000105[0] 1790658078.185324 trunc from 0x91ac85 to 0x077b2b (0x8a3159 bytes) 000107[1] 1790658078.246845 read_OD 0x020000 thru 0x027fff (0x8000 bytes) 000108[1] 1790658078.310959 mapread 0x06b000 thru 0x076899 (0xb89a bytes) 000111[0] 1790658078.371697 mapwrite 0x60b000 thru 0x61a4d0 (0xf4d1 bytes) 000113[1] 1790658078.605392 read_OD 0x324000 thru 0x324fff (0x1000 bytes) 000116[0] 1790658078.654519 punch from 0x07d0fd to 0x08c2d2 (0xf1d6 bytes) 000118[0] 1790658078.734415 mapwrite 0x8e1000 thru 0x8e14de (0x4df bytes) 000119[0] 1790658078.998260 mapwrite 0x584000 thru 0x588738 (0x4739 bytes) 000122[1] 1790658079.241250 read_OD 0x8d7000 thru 0x8d7fff (0x1000 bytes) 000123[0] 1790658079.296986 trunc from 0x8e14df to 0x829702 (0xb7ddc bytes) 000124[0] 1790658079.383540 mapread 0x263000 thru 0x26f9f8 (0xc9f9 bytes) 000125[1] 1790658079.494371 write_OD 0x06c000 thru 0x073fff (0x8000 bytes) 000127[0] 1790658079.555308 mapwrite 0x44d000 thru 0x452d4d (0x5d4e bytes) 000128[0] 1790658079.666275 read 0x1b3000 thru 0x1b8efd (0x5efe bytes) 000129[0] 1790658079.694245 mapread 0x033000 thru 0x03e7c0 (0xb7c1 bytes) 000131[1] 1790658079.732297 mapread 0x29f000 thru 0x2a44b9 (0x54ba bytes) 000132[0] 1790658079.769733 trunc from 0x829703 to 0x794f00 (0x94802 bytes) 000134[1] 1790658079.798369 mapwrite 0x7ed000 thru 0x7f098f (0x3990 bytes) 000135[1] 1790658079.882593 write_OD 0x922000 thru 0x925fff (0x4000 bytes) 000136[0] 1790658079.936661 mapwrite 0x9dc000 thru 0x9dd4cf (0x14d0 bytes) 000137[0] 1790658080.039397 mapwrite 0x881000 thru 0x88dc5d (0xcc5e bytes) 000138[0] 1790658080.105733 write 0x9fe000 thru 0x9ff992 (0x1993 bytes) 000140[0] 1790658080.138927 mapread 0x994000 thru 0x9a34ed (0xf4ee bytes) 000143[1] 1790658080.225664 mapwrite 0x13b000 thru 0x13c92e (0x192f bytes) 000144[1] 1790658080.348702 punch from 0x21b400 to 0x21b9e1 (0x5e2 bytes) 000145[1] 1790658080.371184 read_OD 0x7ee000 thru 0x7f3fff (0x6000 bytes) 000146[1] 1790658080.388162 trunc from 0x9ff993 to 0x6c7601 (0x338391 bytes) 000147[0] 1790658080.418401 mapread 0x065000 thru 0x06fcc9 (0xacca bytes) 000148[1] 1790658080.454493 mapread 0x2dd000 thru 0x2e9662 (0xc663 bytes) 000149[0] 1790658080.491409 mapwrite 00000000 thru 0x0025c9 (0x25ca bytes) 000150[1] 1790658080.555948 trunc from 0x6c7602 to 0x46dace (0x259b33 bytes) 000151[0] 1790658080.625488 mapwrite 0x935000 thru 0x9432dd (0xe2de bytes) truncating to largest ever: 0x949821 000155[1] 1790658080.877076 trunc from 0x9432de to 0x949820 (0x6543 bytes) 000157[0] 1790658080.940422 mapread 0x428000 thru 0x435235 (0xd236 bytes) 000158[1] 1790658080.992516 punch from 0x202dae to 0x2119c9 (0xec1c bytes) 000159[1] 1790658081.027700 trunc from 0x949821 to 0x15f28d (0x7ea593 bytes) 000162[1] 1790658081.048910 mapread 0x04a000 thru 0x052cca (0x8ccb bytes) 000165[1] 1790658081.135266 punch from 0x147d2b to 0x14cbb5 (0x4e8b bytes) 000166[1] 1790658081.161226 read_OD 0x06c000 thru 0x06efff (0x3000 bytes) 000167[0] 1790658081.185199 mapwrite 0x018000 thru 0x02604e (0xe04f bytes) 000170[1] 1790658081.346733 write_OD 0x217000 thru 0x224fff (0xe000 bytes) 000171[1] 1790658081.431711 punch from 0x186cb0 to 0x1922f1 (0xb642 bytes) 000172[0] 1790658081.475170 punch from 0x009c36 to 0x012178 (0x8543 bytes) 000173[0] 1790658081.529194 mapread 0x202000 thru 0x206967 (0x4968 bytes) 000174[1] 1790658081.553842 write_OD 0x213000 thru 0x214fff (0x2000 bytes) 000175[1] 1790658081.740872 read_OD 0x0bd000 thru 0x0c8fff (0xc000 bytes) 000176[0] 1790658081.757927 mapread 0x0bd000 thru 0x0c634d (0x934e bytes) 000177[1] 1790658081.780272 punch from 0x0dafcd to 0x0e755d (0xc591 bytes) 000178[1] 1790658081.837412 mapread 0x071000 thru 0x07684a (0x584b bytes) 000180[1] 1790658081.993309 mapread 0x21a000 thru 0x22427c (0xa27d bytes) 000181[0] 1790658081.995130 trunc from 0x225000 to 0x6d412e (0x4af12f bytes) 000184[0] 1790658082.169203 mapwrite 0x385000 thru 0x38adb4 (0x5db5 bytes) 000187[1] 1790658082.270437 mapwrite 0x032000 thru 0x034012 (0x2013 bytes) 000188[0] 1790658082.385333 mapwrite 0x14b000 thru 0x14b9b0 (0x9b1 bytes) 000189[1] 1790658082.459851 trunc from 0x6d412f to 0x8e4b19 (0x2109eb bytes) 000191[0] 1790658082.523703 mapwrite 0x281000 thru 0x28dd28 (0xcd29 bytes) 000193[0] 1790658082.639621 write 0x639000 thru 0x6454af (0xc4b0 bytes) 000194[0] 1790658082.655475 read 0x69d000 thru 0x69febc (0x2ebd bytes) 000198[1] 1790658082.674081 punch from 0x1322ef to 0x13605d (0x3d6f bytes) 000199[0] 1790658082.699679 punch from 0x7ae75f to 0x7af214 (0xab6 bytes) 000201[1] 1790658082.710696 trunc from 0x8e4b1a to 0x7c8898 (0x11c281 bytes) 000202[0] 1790658082.741987 mapread 0x6a8000 thru 0x6b0120 (0x8121 bytes) 000203[1] 1790658082.777369 write_OD 0x50f000 thru 0x512fff (0x4000 bytes) 000204[0] 1790658082.830205 punch from 0x58af4c to 0x591103 (0x61b8 bytes) 000205[0] 1790658082.875707 trunc from 0x7c8899 to 0x66249a (0x1663fe bytes) 000206[0] 1790658082.909330 trunc from 0x66249b to 0x899fea (0x237b50 bytes) 000207[1] 1790658082.924532 trunc from 0x899feb to 0x31cd8d (0x57d25d bytes) 000209[0] 1790658082.974895 punch from 0x0cc8c1 to 0x0d1654 (0x4d94 bytes) 000210[1] 1790658083.012489 read_OD 0x1bd000 thru 0x1c6fff (0xa000 bytes) 000214[0] 1790658083.066316 mapwrite 0x67c000 thru 0x687d41 (0xbd42 bytes) 000215[0] 1790658083.170246 read 0x5b7000 thru 0x5bc844 (0x5845 bytes) 000216[0] 1790658083.183395 mapwrite 0x66d000 thru 0x67c541 (0xf542 bytes) 000217[1] 1790658083.241867 read_OD 0x637000 thru 0x639fff (0x3000 bytes) 000219[1] 1790658083.263194 trunc from 0x687d42 to 0x349b41 (0x33e200 bytes) 000220[1] 1790658083.312425 write_OD 0x137000 thru 0x13afff (0x4000 bytes) 000222[1] 1790658083.388242 mapread 0x263000 thru 0x271865 (0xe866 bytes) 000223[0] 1790658083.412523 read 0x185000 thru 0x1922e4 (0xd2e5 bytes) 000224[0] 1790658083.457468 trunc from 0x349b42 to 0x3d5833 (0x8bcf2 bytes) 000225[1] 1790658083.503269 trunc from 0x3d5834 to 0x0f175c (0x2e40d7 bytes) 000226[1] 1790658083.645789 trunc from 0x0f175d to 0x942deb (0x85168f bytes) 000227[1] 1790658083.707219 trunc from 0x942dec to 0x919973 (0x29478 bytes) 000228[1] 1790658083.723966 mapread 0x307000 thru 0x30e4e3 (0x74e4 bytes) 000230[1] 1790658083.762293 write_OD 0x7b1000 thru 0x7b9fff (0x9000 bytes) 000233[0] 1790658083.935011 trunc from 0x919974 to 0x5f1289 (0x3286ea bytes) 000235[0] 1790658083.974724 read 0x334000 thru 0x3430ee (0xf0ef bytes) 000236[0] 1790658084.031428 write 0x6f5000 thru 0x6fb661 (0x6662 bytes) 000237[0] 1790658084.032749 read 0x590000 thru 0x591707 (0x1708 bytes) 000239[1] 1790658084.047185 trunc from 0x6fb662 to 0x89d8ed (0x1a228c bytes) 000244[0] 1790658084.229001 mapwrite 0x2fc000 thru 0x300183 (0x4184 bytes) 000245[1] 1790658084.352356 mapwrite 0x122000 thru 0x12d5a1 (0xb5a2 bytes) 000246[0] 1790658084.458444 read 0x253000 thru 0x25602c (0x302d bytes) 000247[1] 1790658084.475970 mapwrite 0x0c9000 thru 0x0d0292 (0x7293 bytes) 000248[0] 1790658084.620070 write 0x388000 thru 0x38b8e3 (0x38e4 bytes) 000249[1] 1790658084.652065 write_OD 0x2a8000 thru 0x2b2fff (0xb000 bytes) 000251[1] 1790658084.712178 mapread 0x402000 thru 0x410979 (0xe97a bytes) 000252[0] 1790658084.761866 write 0x9b6000 thru 0x9ba250 (0x4251 bytes) 000253[1] 1790658084.776451 mapwrite 0x433000 thru 0x43afa0 (0x7fa1 bytes) 000254[1] 1790658084.858027 trunc from 0x9ba251 to 0x5e3838 (0x3d6a18 bytes) 000256[1] 1790658084.896661 mapread 0x181000 thru 0x18708d (0x608e bytes) 000257[1] 1790658084.918886 mapwrite 0x65b000 thru 0x65d808 (0x2809 bytes) 000259[1] 1790658085.002000 mapwrite 0x9c2000 thru 0x9d0661 (0xe662 bytes) 000261[0] 1790658085.178592 read 0x9a3000 thru 0x9b2c97 (0xfc98 bytes) 000262[1] 1790658085.206907 read 0x8b0000 thru 0x8be98a (0xe98b bytes) 000265[1] 1790658085.215869 mapwrite 0x8e6000 thru 0x8f0862 (0xa863 bytes) 000266[1] 1790658085.286346 mapread 0x10d000 thru 0x11ce2f (0xfe30 bytes) 000267[0] 1790658085.347270 mapwrite 0x3c0000 thru 0x3c0176 (0x177 bytes) 000268[0] 1790658085.414189 mapwrite 0x924000 thru 0x92a4aa (0x64ab bytes) 000269[1] 1790658085.490391 write 0x3c2000 thru 0x3c2505 (0x506 bytes) 000270[1] 1790658085.506158 read 0x39b000 thru 0x39c20c (0x120d bytes) 000273[1] 1790658085.527195 mapread 0x991000 thru 0x993b25 (0x2b26 bytes) 000274[0] 1790658085.557968 punch from 0x192db8 to 0x195e42 (0x308b bytes) 000275[1] 1790658085.589345 punch from 0x11e51b to 0x12cca6 (0xe78c bytes) 000276[1] 1790658085.617174 read 0x3c0000 thru 0x3cae28 (0xae29 bytes) 000277[1] 1790658085.650107 read 0x4bb000 thru 0x4bc6fe (0x16ff bytes) 000278[1] 1790658085.673205 write 0x1de000 thru 0x1e7762 (0x9763 bytes) 000279[0] 1790658085.682403 write 0x4c9000 thru 0x4d4151 (0xb152 bytes) 000280[1] 1790658085.737417 punch from 0x8f41e9 to 0x8fb0bc (0x6ed4 bytes) 000281[1] 1790658085.769654 mapread 0x1f5000 thru 0x204b42 (0xfb43 bytes) 000282[0] 1790658085.824068 mapread 0x0f3000 thru 0x0f912f (0x6130 bytes) 000285[1] 1790658085.868367 mapwrite 0x404000 thru 0x40902d (0x502e bytes) 000286[1] 1790658085.982362 punch from 0x2cdb7a to 0x2d4f8b (0x7412 bytes) 000287[1] 1790658086.007638 read 0x738000 thru 0x743abf (0xbac0 bytes) 000288[1] 1790658086.039086 punch from 0x2f3590 to 0x3015fb (0xe06c bytes) 000289[1] 1790658086.044313 write 0x7e0000 thru 0x7ef35a (0xf35b bytes) 000291[1] 1790658086.052163 trunc from 0x9d0662 to 0x5814d1 (0x44f190 bytes) 000292[0] 1790658086.100357 read 0x19a000 thru 0x1a8fb9 (0xefba bytes) 000295[1] 1790658086.130254 read 0x201000 thru 0x20f71c (0xe71d bytes) 000296[0] 1790658086.143951 read 0x48c000 thru 0x497580 (0xb581 bytes) 000297[1] 1790658086.152191 punch from 0x56f7ae to 0x57d2bb (0xdb0e bytes) 000298[1] 1790658086.190400 mapwrite 0x9eb000 thru 0x9eb6d5 (0x6d6 bytes) 000299[0] 1790658086.307952 punch from 0x66e240 to 0x66f812 (0x15d3 bytes) 000300[0] 1790658086.322274 punch from 0x0cc4be to 0x0cca48 (0x58b bytes) 000304[1] 1790658086.371365 punch from 0x1125fe to 0x115097 (0x2a9a bytes) 000305[1] 1790658086.393373 mapwrite 0x7ba000 thru 0x7c8d24 (0xed25 bytes) 000309[0] 1790658086.495295 punch from 0x394bf5 to 0x39bb8b (0x6f97 bytes) 000311[0] 1790658086.532244 read 0x130000 thru 0x13e815 (0xe816 bytes) 000313[0] 1790658086.550154 mapread 0x3e3000 thru 0x3e9527 (0x6528 bytes) 000314[0] 1790658086.573386 read 0x100000 thru 0x104808 (0x4809 bytes) 000315[1] 1790658086.583819 mapwrite 0x6b8000 thru 0x6c3121 (0xb122 bytes) 000317[1] 1790658086.709913 trunc from 0x9eb6d6 to 0x4fa8a3 (0x4f0e32 bytes) 000318[0] 1790658086.733638 write 0x6fc000 thru 0x70ae4a (0xee4b bytes) 000319[1] 1790658086.770639 punch from 0x0c4f8e to 0x0cd26a (0x82dd bytes) 000320[1] 1790658086.819334 read 0x200000 thru 0x205a12 (0x5a13 bytes) 000324[0] 1790658086.858213 trunc from 0x70ae4b to 0x65f4aa (0xab9a0 bytes) 000325[0] 1790658086.903115 read 0x37f000 thru 0x37fadd (0xade bytes) 000326[0] 1790658086.935848 write 0x8fc000 thru 0x90bbd5 (0xfbd6 bytes) 000327[0] 1790658086.938649 read 0x1d0000 thru 0x1db0e1 (0xb0e2 bytes) 000328[1] 1790658086.971438 read 0x242000 thru 0x24d4ff (0xb500 bytes) 000329[1] 1790658087.011261 trunc from 0x90bbd6 to 0x707bbe (0x204017 bytes) 000331[1] 1790658087.071858 punch from 0x278afc to 0x28618d (0xd692 bytes) 000332[1] 1790658087.093216 punch from 0x4c129d to 0x4cc240 (0xafa4 bytes) 000333[0] 1790658087.098624 trunc from 0x707bbf to 0x8d59c5 (0x1cde07 bytes) 000334[0] 1790658087.162571 mapread 0x86e000 thru 0x87287c (0x487d bytes) 000335[0] 1790658087.191091 mapwrite 0x343000 thru 0x3451e5 (0x21e6 bytes) 000337[0] 1790658087.268323 punch from 0x0d65c6 to 0x0e26ec (0xc127 bytes) 000338[1] 1790658087.286066 mapread 0x74b000 thru 0x751ef3 (0x6ef4 bytes) 000339[0] 1790658087.329735 punch from 0x65d336 to 0x664db8 (0x7a83 bytes) 000340[0] 1790658087.344736 trunc from 0x8d59c6 to 0x482fa3 (0x452a22 bytes) 000341[0] 1790658087.370563 write 0x9ff000 thru 0x9ff5b3 (0x5b4 bytes) 000343[0] 1790658087.424596 mapread 0x022000 thru 0x02c723 (0xa724 bytes) 000346[0] 1790658087.472928 mapread 0x42e000 thru 0x43a152 (0xc153 bytes) 000349[0] 1790658087.530153 punch from 0x8efb2f to 0x8f33f1 (0x38c3 bytes) 000350[0] 1790658087.538455 punch from 0x72bf6e to 0x734842 (0x88d5 bytes) 000352[0] 1790658087.555509 punch from 0x6ef0c2 to 0x6fc34e (0xd28d bytes) 000353[0] 1790658087.561688 trunc from 0x9ff5b4 to 0x4d7977 (0x527c3c bytes) 000354[1] 1790658087.585151 mapread 0x46b000 thru 0x476268 (0xb269 bytes) 000355[1] 1790658087.644565 write 0x763000 thru 0x76a027 (0x7028 bytes) 000356[1] 1790658087.670086 mapread 0x1f8000 thru 0x1febec (0x6bed bytes) 000357[0] 1790658087.691407 write 0x772000 thru 0x77c55e (0xa55f bytes) 000359[0] 1790658087.711899 trunc from 0x77c55f to 0x5bae5d (0x1c1701 bytes) 000361[1] 1790658087.736869 write 0x0b9000 thru 0x0bb517 (0x2518 bytes) 000365[0] 1790658087.790398 trunc from 0x5bae5e to 0x5efdcf (0x34f72 bytes) 000367[0] 1790658087.847447 mapwrite 0x9e4000 thru 0x9f3fff (0x10000 bytes) 000368[1] 1790658087.976691 write 0x95e000 thru 0x963d22 (0x5d23 bytes) 000371[1] 1790658088.039561 read 0x771000 thru 0x77498e (0x398f bytes) 000372[0] 1790658088.050850 mapwrite 0x628000 thru 0x634983 (0xc984 bytes) 000373[0] 1790658088.251851 mapread 0x1fa000 thru 0x202272 (0x8273 bytes) 000374[0] 1790658088.273698 trunc from 0x9f4000 to 0x498448 (0x55bbb7 bytes) 000376[1] 1790658088.307109 punch from 0x3e2edd to 0x3ed45c (0xa580 bytes) 000381[1] 1790658088.382669 mapread 0x46c000 thru 0x478431 (0xc432 bytes) 000383[0] 1790658088.450705 mapwrite 0x844000 thru 0x852121 (0xe122 bytes) 000387[0] 1790658088.591638 read 0x1b5000 thru 0x1b9c03 (0x4c04 bytes) 000389[1] 1790658088.612266 read 0x0df000 thru 0x0e21d1 (0x31d2 bytes) 000391[0] 1790658088.643906 punch from 0x5d0973 to 0x5deed9 (0xe567 bytes) 000393[0] 1790658088.722839 write 0x2d1000 thru 0x2d586e (0x486f bytes) 000394[0] 1790658088.744421 mapwrite 0x6e5000 thru 0x6f4609 (0xf60a bytes) 000396[0] 1790658088.968323 read 0x299000 thru 0x2a874b (0xf74c bytes) 000397[1] 1790658089.031673 mapread 0x602000 thru 0x60adb2 (0x8db3 bytes) 000402[0] 1790658089.141924 mapread 0x1f8000 thru 0x20656e (0xe56f bytes) 000403[1] 1790658089.178220 write 0x333000 thru 0x338f55 (0x5f56 bytes) 000405[0] 1790658089.216485 read 0x4a2000 thru 0x4a6821 (0x4822 bytes) 000407[0] 1790658089.252806 trunc from 0x852122 to 0x3d1b8a (0x480597 bytes) 000408[1] 1790658089.304298 mapread 0x18e000 thru 0x1942c8 (0x62c9 bytes) 000409[0] 1790658089.384719 mapread 0x17c000 thru 0x18892f (0xc930 bytes) 000410[1] 1790658089.421826 mapwrite 0x56f000 thru 0x574605 (0x5606 bytes) 000411[1] 1790658089.608708 mapwrite 0x07e000 thru 0x0816f1 (0x36f2 bytes) 000412[0] 1790658089.717673 mapread 0x0fb000 thru 0x104e0e (0x9e0f bytes) 000414[0] 1790658089.770706 punch from 0x1fd625 to 0x2077cd (0xa1a9 bytes) 000417[1] 1790658089.789878 mapwrite 0x603000 thru 0x607737 (0x4738 bytes) 000418[1] 1790658089.978395 trunc from 0x607738 to 0x573a60 (0x93cd7 bytes) 000419[1] 1790658089.989911 trunc from 0x573a61 to 0x4ae9d6 (0xc508a bytes) 000421[1] 1790658090.017314 read 0x434000 thru 0x43ae0a (0x6e0b bytes) 000422[0] 1790658090.026731 mapread 0x475000 thru 0x476577 (0x1578 bytes) 000423[1] 1790658090.043034 write 0x0b6000 thru 0x0c0516 (0xa517 bytes) 000424[1] 1790658090.056142 mapwrite 0x491000 thru 0x496cd2 (0x5cd3 bytes) 000425[0] 1790658090.154130 mapwrite 0x535000 thru 0x544d2e (0xfd2f bytes) 000426[1] 1790658090.297235 punch from 0x4cc6df to 0x4cc7d4 (0xf6 bytes) 000427[0] 1790658090.331303 trunc from 0x544d2f to 0x7fd721 (0x2b89f3 bytes) 000428[0] 1790658090.402055 mapread 0x002000 thru 0x00f8d7 (0xd8d8 bytes) 000430[0] 1790658090.566877 read 0x10e000 thru 0x112e84 (0x4e85 bytes) 000431[1] 1790658090.613162 punch from 0x02bd67 to 0x0319ac (0x5c46 bytes) 000432[1] 1790658090.657606 punch from 0x6b3b9f to 0x6b6779 (0x2bdb bytes) 000435[0] 1790658090.690387 trunc from 0x7fd722 to 0x81b681 (0x1df60 bytes) 000436[1] 1790658090.742131 mapwrite 0x396000 thru 0x3977f7 (0x17f8 bytes) 000437[1] 1790658090.840509 punch from 0x59a704 to 0x59c5f6 (0x1ef3 bytes) 000439[1] 1790658090.868069 punch from 0x5c8924 to 0x5cdcf3 (0x53d0 bytes) 000445[1] 1790658090.892545 trunc from 0x81b682 to 0x6514bb (0x1ca1c6 bytes) 000446[0] 1790658090.918923 read 0x1bd000 thru 0x1bfa48 (0x2a49 bytes) 000447[1] 1790658090.939618 punch from 0x4b9464 to 0x4bac24 (0x17c1 bytes) 000449[1] 1790658090.996859 punch from 0x302689 to 0x3099b4 (0x732c bytes) 000450[1] 1790658091.018651 punch from 0x45073e to 0x45ab51 (0xa414 bytes) 000451[0] 1790658091.026890 mapwrite 0x291000 thru 0x2a0cf6 (0xfcf7 bytes) 000452[0] 1790658091.154139 punch from 0x2cce45 to 0x2d25d0 (0x578c bytes) truncating to largest ever: 0x9c460e 000453[1] 1790658091.158383 trunc from 0x6514bc to 0x9c460d (0x373152 bytes) 000455[1] 1790658091.223605 punch from 0x2399e3 to 0x23ac9b (0x12b9 bytes) 000456[0] 1790658091.230223 mapread 0x667000 thru 0x667d03 (0xd04 bytes) 000459[0] 1790658091.258528 trunc from 0x9c460e to 0x8c115f (0x1034ae bytes) 000460[0] 1790658091.324250 punch from 0x6855d6 to 0x68b6a5 (0x60d0 bytes) 000461[0] 1790658091.353625 trunc from 0x8c1160 to 0x32c5a9 (0x594bb6 bytes) 000465[1] 1790658091.467481 punch from 0x079266 to 0x081f6b (0x8d06 bytes) 000467[1] 1790658091.486236 mapwrite 0x1b7000 thru 0x1c5a9d (0xea9e bytes) 000471[0] 1790658091.700706 read 0x117000 thru 0x11a189 (0x318a bytes) 000474[1] 1790658091.831136 mapwrite 0x1bc000 thru 0x1c5a80 (0x9a81 bytes) 000475[1] 1790658092.008072 mapread 0x203000 thru 0x21110c (0xe10d bytes) 000476[1] 1790658092.042938 trunc from 0x32c5aa to 0x98011d (0x653b74 bytes) 000477[0] 1790658092.134520 write 0x487000 thru 0x49164c (0xa64d bytes) 000478[1] 1790658092.193282 trunc from 0x98011e to 0x2f8c86 (0x687497 bytes) 000480[0] 1790658092.239491 punch from 0x2d20c8 to 0x2e0320 (0xe259 bytes) 000481[1] 1790658092.254840 trunc from 0x2f8c87 to 0x3a7a2b (0xaeda5 bytes) 000482[0] 1790658092.280242 mapread 0x17c000 thru 0x1899f9 (0xd9fa bytes) 000484[1] 1790658092.383334 trunc from 0x3a7a2c to 0x65ad0f (0x2b32e4 bytes) 000487[1] 1790658092.520410 read 0x40d000 thru 0x417ffe (0xafff bytes) 000490[1] 1790658092.592689 mapwrite 0x8d8000 thru 0x8e18bd (0x98be bytes) 000491[0] 1790658092.707603 punch from 0x174a8c to 0x18441e (0xf993 bytes) 000492[1] 1790658092.749058 trunc from 0x8e18be to 0x40ce1a (0x4d4aa3 bytes) 000494[1] 1790658092.779631 write 0x651000 thru 0x654051 (0x3052 bytes) 000496[1] 1790658092.810936 punch from 0x007bb4 to 0x01021b (0x8668 bytes) 000497[0] 1790658092.830298 mapwrite 0x032000 thru 0x03e70b (0xc70c bytes) All operations completed A-OK! PASS 16a (55s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:01:38 (1790658098) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.377004 s, 22.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.352925 s, 23.8 MB/s Chance of close/open is 1 in 50 Seed set to 8102 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x95ae59 000002[1] 1790658103.084148 trunc from 00000000 to 0x95ae58 (0x95ae59 bytes) 000003[1] 1790658103.152380 read 0x26577f thru 0x265816 (0x98 bytes) 000004[0] 1790658103.191017 write 0x257b29 thru 0x258ed9 (0x13b1 bytes) 000005[0] 1790658103.263568 mapwrite 0x5ba7af thru 0x5bb5f1 (0xe43 bytes) 000006[0] 1790658103.486566 read 0x1808a0 thru 0x1809bf (0x120 bytes) 000007[0] 1790658103.491299 punch from 0x2e4176 to 0x2e5f62 (0x1ded bytes) 000010[0] 1790658103.526574 mapread 0x8f835c thru 0x8f84b3 (0x158 bytes) 000012[1] 1790658103.540090 read 0x4d8269 thru 0x4d9582 (0x131a bytes) 000013[0] 1790658103.573073 mapwrite 0x4c25d0 thru 0x4c3643 (0x1074 bytes) 000014[1] 1790658103.717607 mapread 0x85ce09 thru 0x85d33a (0x532 bytes) 000015[1] 1790658103.755155 mapread 0x6bc0b7 thru 0x6bc8f7 (0x841 bytes) 000016[1] 1790658103.782503 write 0x0f456d thru 0x0f61d7 (0x1c6b bytes) 000019[1] 1790658103.822558 write 0x3ff1a9 thru 0x400db9 (0x1c11 bytes) 000022[1] 1790658103.904670 trunc from 0x95ae59 to 0x82bf61 (0x12eef7 bytes) 000023[0] 1790658103.949296 mapread 0x6639db thru 0x664a81 (0x10a7 bytes) 000025[0] 1790658103.994571 trunc from 0x82bf62 to 0x29ba3e (0x590523 bytes) 000026[0] 1790658104.067024 mapread 0x1fd332 thru 0x1fe8bd (0x158c bytes) 000028[1] 1790658104.105251 punch from 0x152ed0 to 0x153385 (0x4b6 bytes) 000031[0] 1790658104.160637 write 0x18fc83 thru 0x19001d (0x39b bytes) 000032[1] 1790658104.198185 write 0x7c5e5a thru 0x7c612c (0x2d3 bytes) 000037[0] 1790658104.286449 mapwrite 0x9ef240 thru 0x9f0742 (0x1503 bytes) 000040[1] 1790658104.473411 mapread 0x11934f thru 0x11a10c (0xdbe bytes) 000042[0] 1790658104.508210 read 0x95a803 thru 0x95c7d7 (0x1fd5 bytes) 000043[1] 1790658104.513137 read 0x49b9db thru 0x49bb03 (0x129 bytes) 000044[1] 1790658104.529607 mapwrite 0x8bf71b thru 0x8c060d (0xef3 bytes) 000047[1] 1790658104.608397 trunc from 0x9f0743 to 0x28e6d1 (0x762071 bytes) 000050[0] 1790658104.662040 write 0x5a604e thru 0x5a70c9 (0x107c bytes) 000051[0] 1790658104.685118 read 0x58900f thru 0x58afdb (0x1fcd bytes) 000052[1] 1790658104.703514 mapread 0x318329 thru 0x319ed4 (0x1bac bytes) 000053[1] 1790658104.749480 punch from 0x343720 to 0x345611 (0x1ef2 bytes) 000055[1] 1790658104.798635 mapwrite 0x6ed1df thru 0x6edd49 (0xb6b bytes) 000056[0] 1790658104.902066 read 0x6e4d6e thru 0x6e6688 (0x191b bytes) 000057[0] 1790658104.934492 write 0x6241a0 thru 0x625b2c (0x198d bytes) 000059[0] 1790658104.984264 mapread 0x4634d4 thru 0x463d6d (0x89a bytes) 000060[0] 1790658105.018688 write 0x733013 thru 0x734aaa (0x1a98 bytes) 000061[1] 1790658105.038451 trunc from 0x734aab to 0x168e3f (0x5cbc6b bytes) 000062[1] 1790658105.098556 mapread 0x164cee thru 0x164e37 (0x14a bytes) 000063[0] 1790658105.127429 punch from 0x0012e6 to 0x001f17 (0xc32 bytes) 000065[0] 1790658105.176081 read 0x0e602a thru 0x0e6452 (0x429 bytes) 000066[1] 1790658105.233673 read 0x0c7473 thru 0x0c8362 (0xef0 bytes) 000067[1] 1790658105.287700 punch from 0x05dd49 to 0x05ebfa (0xeb2 bytes) 000069[0] 1790658105.321197 punch from 0x0fb7e8 to 0x0fbe1f (0x638 bytes) 000070[1] 1790658105.348218 mapwrite 0x232847 thru 0x233701 (0xebb bytes) 000071[1] 1790658105.445226 trunc from 0x233702 to 0x3f6d40 (0x1c363f bytes) 000072[1] 1790658105.467513 write 0x3ba66d thru 0x3babe4 (0x578 bytes) 000073[1] 1790658105.499446 punch from 0x0ae70c to 0x0afa8c (0x1381 bytes) 000074[1] 1790658105.506162 mapwrite 0x1f1d51 thru 0x1f231c (0x5cc bytes) 000079[1] 1790658105.595110 read 0x2090ac thru 0x2096ff (0x654 bytes) 000085[0] 1790658105.654015 mapwrite 0x188f69 thru 0x18922d (0x2c5 bytes) 000086[1] 1790658105.818517 read 0x362cc4 thru 0x3644af (0x17ec bytes) 000088[0] 1790658105.954402 read 0x2cd9b4 thru 0x2cf629 (0x1c76 bytes) 000089[1] 1790658106.059437 trunc from 0x3f6d41 to 0x28f6eb (0x167655 bytes) 000090[0] 1790658106.117901 mapwrite 0x69a298 thru 0x69b190 (0xef9 bytes) 000091[1] 1790658106.294675 trunc from 0x69b191 to 0x01738b (0x683e05 bytes) 000092[0] 1790658106.362977 read 0x00bf1e thru 0x00daae (0x1b91 bytes) 000095[1] 1790658106.492687 read 0x004647 thru 0x004ae3 (0x49d bytes) 000096[0] 1790658106.570577 read 0x011225 thru 0x0112c0 (0x9c bytes) 000098[1] 1790658106.630906 read 0x011104 thru 0x012d83 (0x1c80 bytes) 000099[0] 1790658106.651201 write 0x855982 thru 0x8574c3 (0x1b42 bytes) 000102[1] 1790658106.739739 trunc from 0x8574c4 to 0x6a9897 (0x1adc2c bytes) 000103[1] 1790658106.774000 mapwrite 0x03e959 thru 0x03fe41 (0x14e9 bytes) 000105[0] 1790658106.854128 mapread 0x2beb5e thru 0x2bedde (0x281 bytes) 000108[0] 1790658106.971572 mapwrite 0x7e5859 thru 0x7e5f54 (0x6fc bytes) 000109[0] 1790658107.135504 trunc from 0x7e5f55 to 0x23cd4b (0x5a9209 bytes) 000111[0] 1790658107.189154 punch from 0x0ac602 to 0x0ac800 (0x1ff bytes) 000112[0] 1790658107.245101 punch from 0x177d2d to 0x177d89 (0x5d bytes) 000113[0] 1790658107.283893 mapwrite 0x6c3a4f thru 0x6c57aa (0x1d5c bytes) 000116[0] 1790658107.480392 mapread 0x0b365b thru 0x0b4ec8 (0x186e bytes) 000118[0] 1790658107.553080 read 0x664ba2 thru 0x6657cc (0xc2b bytes) 000120[0] 1790658107.577984 read 0x134128 thru 0x135a2c (0x1905 bytes) 000122[0] 1790658107.584253 punch from 0x39b18a to 0x39cb2a (0x19a1 bytes) 000125[1] 1790658107.648260 trunc from 0x6c57ab to 0x40db9e (0x2b7c0c bytes) 000130[0] 1790658107.796211 punch from 0x394438 to 0x3947bc (0x385 bytes) 000132[1] 1790658107.846486 mapread 0x1ed5c6 thru 0x1ed8e0 (0x31b bytes) 000134[0] 1790658107.893207 mapwrite 0x8480c1 thru 0x84814e (0x8e bytes) 000135[0] 1790658107.998558 trunc from 0x84814f to 0x2f5d29 (0x552425 bytes) 000136[1] 1790658108.014841 write 0x302dbb thru 0x3032b6 (0x4fc bytes) 000139[0] 1790658108.073669 mapread 0x05ab66 thru 0x05c972 (0x1e0d bytes) 000140[1] 1790658108.155710 mapread 0x0a85c2 thru 0x0aa006 (0x1a45 bytes) 000143[1] 1790658108.188386 mapread 0x1d6ae0 thru 0x1d7331 (0x852 bytes) 000144[0] 1790658108.203679 mapwrite 0x355aed thru 0x3575be (0x1ad2 bytes) 000145[1] 1790658108.346451 punch from 0x2a5214 to 0x2a6cd3 (0x1ac0 bytes) 000146[1] 1790658108.410266 read 0x14cca4 thru 0x14d370 (0x6cd bytes) 000147[1] 1790658108.565573 trunc from 0x3575bf to 0x23dbfb (0x1199c3 bytes) 000148[0] 1790658108.633525 trunc from 0x23dbfc to 0x5c612a (0x38852f bytes) 000149[0] 1790658108.695143 mapwrite 0x2a408f thru 0x2a51bf (0x1131 bytes) 000151[1] 1790658108.779382 trunc from 0x5c612b to 0x1bf2d8 (0x406e52 bytes) 000152[1] 1790658108.826922 punch from 0x0c8d18 to 0x0c955c (0x845 bytes) 000154[0] 1790658108.887916 read 0x035769 thru 0x036f78 (0x1810 bytes) 000155[1] 1790658108.948722 punch from 0x121448 to 0x1227c2 (0x137b bytes) 000157[0] 1790658108.990518 mapwrite 0x3c3a03 thru 0x3c4d92 (0x1390 bytes) 000160[1] 1790658109.152546 mapread 0x2d23d7 thru 0x2d2918 (0x542 bytes) 000161[0] 1790658109.199952 mapwrite 0x84f645 thru 0x85098b (0x1347 bytes) 000162[0] 1790658109.390691 write 0x3286aa thru 0x32a1d5 (0x1b2c bytes) 000163[0] 1790658109.419012 write 0x46d31d thru 0x46ea49 (0x172d bytes) 000165[0] 1790658109.485577 read 0x198e61 thru 0x199581 (0x721 bytes) 000166[1] 1790658109.496599 read 0x752ca9 thru 0x752e1c (0x174 bytes) 000168[1] 1790658109.537034 mapread 0x6e0426 thru 0x6e2363 (0x1f3e bytes) 000170[1] 1790658109.575201 trunc from 0x85098c to 0x598425 (0x2b8566 bytes) 000171[0] 1790658109.637011 write 0x76c521 thru 0x76e0fd (0x1bdd bytes) 000173[0] 1790658109.686949 read 0x441b5c thru 0x441fc1 (0x466 bytes) 000176[1] 1790658109.732699 punch from 0x2fd6e2 to 0x2fe8df (0x11fe bytes) 000178[1] 1790658109.765380 punch from 0x6864c8 to 0x686c52 (0x78b bytes) 000180[0] 1790658109.825970 mapwrite 0x9d2a11 thru 0x9d4947 (0x1f37 bytes) 000182[0] 1790658109.975694 write 0x3d45cb thru 0x3d61c3 (0x1bf9 bytes) 000183[1] 1790658109.993640 write 0x0bd3f2 thru 0x0bda96 (0x6a5 bytes) 000184[1] 1790658110.026658 trunc from 0x9d4948 to 0x1bf46d (0x8154da bytes) 000187[0] 1790658110.121160 read 0x1a1b8a thru 0x1a373f (0x1bb6 bytes) 000188[0] 1790658110.196185 read 0x0a5473 thru 0x0a7444 (0x1fd2 bytes) 000190[0] 1790658110.219434 mapread 0x19a1e0 thru 0x19a440 (0x261 bytes) 000191[1] 1790658110.244112 punch from 0x09e667 to 0x09ebce (0x568 bytes) 000192[0] 1790658110.307709 write 0x1aefba thru 0x1afcbe (0xd05 bytes) 000193[1] 1790658110.404699 read 0x02faba thru 0x0319bf (0x1f06 bytes) 000194[1] 1790658110.472840 trunc from 0x1bf46e to 0x0bc2d1 (0x10319c bytes) 000195[0] 1790658110.513101 punch from 0x0b389c to 0x0b3c37 (0x39c bytes) 000197[1] 1790658110.569634 mapread 0x09d956 thru 0x09e04d (0x6f8 bytes) 000198[0] 1790658110.624328 trunc from 0x0bc2d2 to 0x3fbced (0x33fa1c bytes) 000199[1] 1790658110.688067 mapwrite 0x25c1a6 thru 0x25e155 (0x1fb0 bytes) 000200[0] 1790658110.834865 punch from 0x02a120 to 0x02abd9 (0xaba bytes) 000201[0] 1790658110.884218 write 0x83597b thru 0x837022 (0x16a8 bytes) 000202[1] 1790658110.886325 mapread 0x30f085 thru 0x31037e (0x12fa bytes) 000203[0] 1790658110.950726 mapread 0x3d6849 thru 0x3d84ae (0x1c66 bytes) 000205[0] 1790658111.078323 trunc from 0x837023 to 0x7c2b71 (0x744b1 bytes) 000206[1] 1790658111.155244 read 0x30c319 thru 0x30d96f (0x1657 bytes) 000207[0] 1790658111.190448 mapread 0x0ce2f7 thru 0x0cea8c (0x796 bytes) 000209[0] 1790658111.231704 write 0x20dc0c thru 0x20e4c3 (0x8b8 bytes) 000211[1] 1790658111.264212 punch from 0x11577b to 0x1162ca (0xb50 bytes) 000213[0] 1790658111.318229 write 0x009927 thru 0x00acab (0x1385 bytes) 000216[0] 1790658111.395664 mapread 0x5eff69 thru 0x5f04c0 (0x558 bytes) 000218[1] 1790658111.465508 write 0x54bfd7 thru 0x54d03c (0x1066 bytes) 000220[0] 1790658111.535832 read 0x265af1 thru 0x26642c (0x93c bytes) 000221[1] 1790658111.557541 trunc from 0x7c2b72 to 0x842c1f (0x800ae bytes) 000223[0] 1790658111.699678 punch from 0x26d37c to 0x26dd04 (0x989 bytes) 000227[0] 1790658111.894183 read 0x48970d thru 0x48a611 (0xf05 bytes) 000228[1] 1790658111.975504 read 0x56c16b thru 0x56db67 (0x19fd bytes) 000229[0] 1790658111.989382 write 0x40c6da thru 0x40d0b5 (0x9dc bytes) 000232[0] 1790658112.052706 write 0x3d37a1 thru 0x3d53f8 (0x1c58 bytes) 000235[1] 1790658112.085277 write 0x41386a thru 0x414b5d (0x12f4 bytes) 000237[0] 1790658112.135664 mapwrite 0x2071d3 thru 0x207939 (0x767 bytes) 000238[1] 1790658112.208220 mapread 0x13f028 thru 0x140245 (0x121e bytes) 000239[0] 1790658112.245303 punch from 0x26f0f2 to 0x270764 (0x1673 bytes) 000241[1] 1790658112.325535 mapread 0x8238e6 thru 0x825273 (0x198e bytes) 000243[1] 1790658112.393368 mapwrite 0x68d215 thru 0x68dec4 (0xcb0 bytes) 000245[1] 1790658112.457704 mapread 0x1f1d95 thru 0x1f29e8 (0xc54 bytes) 000246[1] 1790658112.477486 mapread 0x7c705a thru 0x7c8d51 (0x1cf8 bytes) 000248[0] 1790658112.511998 mapwrite 0x9e286e thru 0x9e41b2 (0x1945 bytes) 000249[0] 1790658112.612397 mapwrite 0x96b064 thru 0x96bc67 (0xc04 bytes) 000250[0] 1790658112.667334 write 0x230eb4 thru 0x232b4b (0x1c98 bytes) 000253[0] 1790658112.701648 punch from 0x45c913 to 0x45e1e9 (0x18d7 bytes) 000258[0] 1790658112.790363 punch from 0x28ed31 to 0x28f5c0 (0x890 bytes) 000260[1] 1790658112.840008 mapread 0x17792c thru 0x17900d (0x16e2 bytes) 000264[1] 1790658112.892151 write 0x46b473 thru 0x46d345 (0x1ed3 bytes) 000265[0] 1790658112.910518 trunc from 0x9e41b3 to 0x447a35 (0x59c77d bytes) 000266[1] 1790658112.954985 mapwrite 0x6bd2dd thru 0x6be733 (0x1457 bytes) 000267[0] 1790658113.073637 write 0x5ce14e thru 0x5cf293 (0x1146 bytes) 000272[0] 1790658113.162308 write 0x41cb4f thru 0x41cd26 (0x1d8 bytes) 000277[0] 1790658113.272576 mapwrite 0x61c234 thru 0x61d6ad (0x147a bytes) 000279[1] 1790658113.366419 read 0x5b0f8a thru 0x5b1a06 (0xa7d bytes) 000280[0] 1790658113.397129 mapwrite 0x027766 thru 0x02845c (0xcf7 bytes) 000284[0] 1790658113.530541 mapread 0x391b20 thru 0x39289c (0xd7d bytes) 000285[0] 1790658113.543544 mapread 0x4570b9 thru 0x45861d (0x1565 bytes) 000286[0] 1790658113.554061 write 0x6d8890 thru 0x6d8cfe (0x46f bytes) 000287[0] 1790658113.573517 read 0x6cc8ad thru 0x6cd5c7 (0xd1b bytes) 000289[1] 1790658113.644808 write 0x72366d thru 0x724e27 (0x17bb bytes) 000292[0] 1790658113.711375 mapread 0x048a57 thru 0x049bb5 (0x115f bytes) 000293[1] 1790658113.740229 read 0x580efe thru 0x5821f1 (0x12f4 bytes) 000294[0] 1790658113.763990 mapwrite 0x71a440 thru 0x71af67 (0xb28 bytes) 000295[0] 1790658113.845058 mapwrite 0x33f0d3 thru 0x33f39b (0x2c9 bytes) 000298[1] 1790658113.973964 mapwrite 0x71d3bb thru 0x71ef4a (0x1b90 bytes) 000300[0] 1790658114.092051 read 0x4e0bef thru 0x4e1433 (0x845 bytes) 000302[0] 1790658114.128864 write 0x664abe thru 0x665e48 (0x138b bytes) 000303[0] 1790658114.173193 trunc from 0x724e28 to 0x89aa87 (0x175c60 bytes) 000306[0] 1790658114.228094 punch from 0x708866 to 0x7090b2 (0x84d bytes) 000308[1] 1790658114.277936 mapread 0x2f7976 thru 0x2f8c14 (0x129f bytes) 000309[0] 1790658114.363146 trunc from 0x89aa88 to 0x770b7d (0x129f0a bytes) 000311[1] 1790658114.453558 read 0x352fd6 thru 0x3532c8 (0x2f3 bytes) 000312[0] 1790658114.480362 mapread 0x2e51fc thru 0x2e6de7 (0x1bec bytes) 000317[0] 1790658114.588090 mapread 0x569180 thru 0x569527 (0x3a8 bytes) 000318[1] 1790658114.600454 read 0x24a423 thru 0x24a53c (0x11a bytes) 000319[1] 1790658114.608983 punch from 0x6bbed3 to 0x6bbf2b (0x59 bytes) 000320[0] 1790658114.648016 punch from 0x4a4972 to 0x4a518b (0x81a bytes) 000321[0] 1790658114.684460 mapread 0x67389f thru 0x674a94 (0x11f6 bytes) 000322[0] 1790658114.717155 mapread 0x3caa07 thru 0x3cc8e7 (0x1ee1 bytes) 000323[0] 1790658114.779384 read 0x5335a6 thru 0x534975 (0x13d0 bytes) 000326[1] 1790658114.868213 mapwrite 0x076762 thru 0x077a72 (0x1311 bytes) 000327[0] 1790658115.081829 read 0x3ee70b thru 0x3ef73c (0x1032 bytes) 000328[1] 1790658115.129102 mapwrite 0x9a96fe thru 0x9aa314 (0xc17 bytes) 000329[0] 1790658115.369177 write 0x4c59da thru 0x4c6ac8 (0x10ef bytes) 000330[1] 1790658115.428161 write 0x641be3 thru 0x641e43 (0x261 bytes) 000331[1] 1790658115.460061 write 0x995152 thru 0x996aed (0x199c bytes) 000332[0] 1790658115.471196 read 0x0d91eb thru 0x0d96a4 (0x4ba bytes) 000335[1] 1790658115.577177 mapread 0x5b3414 thru 0x5b470c (0x12f9 bytes) 000336[0] 1790658115.597931 trunc from 0x9aa315 to 0x43cd5d (0x56d5b7 bytes) 000338[0] 1790658115.727465 mapread 0x094d5f thru 0x09636d (0x160f bytes) 000339[0] 1790658115.756483 mapwrite 0x4484d6 thru 0x4498b0 (0x13db bytes) 000340[1] 1790658115.981239 punch from 0x408ca2 to 0x40a1f5 (0x1554 bytes) 000341[0] 1790658116.045392 mapread 0x0e705a thru 0x0e82c5 (0x126c bytes) 000342[0] 1790658116.097482 write 0x1f124b thru 0x1f1d7c (0xb32 bytes) 000347[0] 1790658116.246046 mapread 0x1c63b8 thru 0x1c715c (0xda5 bytes) 000349[1] 1790658116.310663 read 0x25f769 thru 0x260eac (0x1744 bytes) 000350[1] 1790658116.368194 trunc from 0x4498b1 to 0x22d4e7 (0x21c3c9 bytes) 000351[0] 1790658116.417153 write 0x5cbe98 thru 0x5ccd3b (0xea4 bytes) 000352[1] 1790658116.450650 mapwrite 0x8821aa thru 0x8840a5 (0x1efc bytes) 000353[1] 1790658116.568349 trunc from 0x8840a6 to 0x2daf16 (0x5a918f bytes) 000355[0] 1790658116.639375 trunc from 0x2daf17 to 0x6e37fe (0x4088e8 bytes) 000361[1] 1790658116.814147 write 0x77e9da thru 0x78089a (0x1ec1 bytes) 000367[1] 1790658116.921589 mapread 0x3faa06 thru 0x3fb231 (0x82c bytes) 000368[1] 1790658116.951053 punch from 0x72b7ae to 0x72c00b (0x85e bytes) 000369[0] 1790658116.959900 mapread 0x4c26f0 thru 0x4c4056 (0x1967 bytes) 000370[1] 1790658117.022542 punch from 0x731939 to 0x7333ac (0x1a74 bytes) 000371[1] 1790658117.058813 trunc from 0x78089b to 0x69075f (0xf013b bytes) 000372[0] 1790658117.112428 punch from 0x65b524 to 0x65bffe (0xadb bytes) 000373[1] 1790658117.154148 read 0x31049f thru 0x31105b (0xbbd bytes) 000375[0] 1790658117.188399 write 0x0631c0 thru 0x064979 (0x17ba bytes) 000376[0] 1790658117.226270 read 0x4c89e6 thru 0x4c909c (0x6b7 bytes) 000377[1] 1790658117.239357 punch from 0x095d7a to 0x097c6c (0x1ef3 bytes) 000379[1] 1790658117.286537 trunc from 0x690760 to 0x32e2ce (0x362491 bytes) 000381[0] 1790658117.351558 mapread 0x1a1293 thru 0x1a19a9 (0x717 bytes) 000383[1] 1790658117.387820 write 0x531c8a thru 0x531fdd (0x354 bytes) 000384[1] 1790658117.402054 trunc from 0x531fde to 0x31e77d (0x213860 bytes) 000385[0] 1790658117.459915 trunc from 0x31e77e to 0x2db590 (0x431ed bytes) 000386[0] 1790658117.507526 trunc from 0x2db591 to 0x43dc96 (0x162706 bytes) 000387[1] 1790658117.526552 read 0x3ea4b0 thru 0x3ec257 (0x1da8 bytes) 000388[1] 1790658117.630422 mapread 0x18dfe9 thru 0x18e17b (0x193 bytes) 000389[1] 1790658117.652161 mapwrite 0x9cb41b thru 0x9cbdf6 (0x9dc bytes) 000390[1] 1790658117.813955 mapread 0x03fcbf thru 0x040e80 (0x11c2 bytes) 000392[1] 1790658117.854309 read 0x7ddc05 thru 0x7de81e (0xc1a bytes) 000395[0] 1790658117.881581 read 0x833e74 thru 0x834c6b (0xdf8 bytes) 000397[0] 1790658117.920009 mapread 0x16b065 thru 0x16ba36 (0x9d2 bytes) 000400[1] 1790658117.957049 punch from 0x28c4e7 to 0x28d00e (0xb28 bytes) 000401[1] 1790658117.973458 read 0x6bd3fe thru 0x6bf359 (0x1f5c bytes) 000402[0] 1790658117.974001 read 0x0c4a55 thru 0x0c56d4 (0xc80 bytes) 000403[1] 1790658117.998261 punch from 0x449882 to 0x449997 (0x116 bytes) 000404[1] 1790658118.014460 punch from 0x580455 to 0x58152b (0x10d7 bytes) 000405[0] 1790658118.043975 trunc from 0x9cbdf7 to 0x1ec586 (0x7df870 bytes) 000406[1] 1790658118.116990 mapwrite 0x6fdb8e thru 0x6fecfb (0x116e bytes) 000407[0] 1790658118.202108 mapwrite 0x613d03 thru 0x615558 (0x1856 bytes) 000410[1] 1790658118.331325 trunc from 0x6fecfc to 0x512e60 (0x1ebe9b bytes) 000412[0] 1790658118.369314 mapwrite 0x5e41fd thru 0x5e5197 (0xf9b bytes) 000414[0] 1790658118.487138 mapread 0x59c2ed thru 0x59d2d2 (0xfe6 bytes) 000415[0] 1790658118.499612 read 0x0e0d84 thru 0x0e240f (0x168c bytes) 000417[0] 1790658118.505992 trunc from 0x5e5198 to 0x4b3f9f (0x1311f8 bytes) 000418[1] 1790658118.528588 read 0x1c41bd thru 0x1c471b (0x55f bytes) 000421[0] 1790658118.585615 read 0x45a1a8 thru 0x45be8e (0x1ce7 bytes) 000422[0] 1790658118.597335 mapread 0x36fc13 thru 0x370d94 (0x1182 bytes) 000423[1] 1790658118.606543 mapread 0x391eb2 thru 0x3939c5 (0x1b14 bytes) 000424[0] 1790658118.636796 write 0x2325ee thru 0x234467 (0x1e7a bytes) 000425[1] 1790658118.699273 punch from 0x3d38a3 to 0x3d4f75 (0x16d3 bytes) 000427[1] 1790658118.776331 mapwrite 0x5fdb35 thru 0x5ff221 (0x16ed bytes) 000428[1] 1790658118.956375 trunc from 0x5ff222 to 0x930f87 (0x331d66 bytes) 000429[0] 1790658119.004681 write 0x21d9f9 thru 0x21e868 (0xe70 bytes) 000430[0] 1790658119.113057 trunc from 0x930f88 to 0x8c80e1 (0x68ea6 bytes) 000431[1] 1790658119.186949 read 0x45c45e thru 0x45d83f (0x13e2 bytes) 000434[1] 1790658119.218115 read 0x617ad1 thru 0x619595 (0x1ac5 bytes) 000435[1] 1790658119.225557 mapwrite 0x3d96b5 thru 0x3da57e (0xeca bytes) 000436[1] 1790658119.324699 punch from 0x4ed9b3 to 0x4ee438 (0xa86 bytes) 000439[1] 1790658119.404504 write 0x39a874 thru 0x39c620 (0x1dad bytes) 000441[1] 1790658119.457686 punch from 0x5e745c to 0x5e885e (0x1403 bytes) 000442[0] 1790658119.462748 read 0x740ac5 thru 0x741865 (0xda1 bytes) 000445[1] 1790658119.525290 punch from 0x4a962d to 0x4a9751 (0x125 bytes) 000447[1] 1790658119.546507 read 0x793997 thru 0x795976 (0x1fe0 bytes) 000448[1] 1790658119.552971 punch from 0x4dccb0 to 0x4dd94f (0xca0 bytes) 000449[0] 1790658119.559009 trunc from 0x8c80e2 to 0x1a6a21 (0x7216c0 bytes) 000450[0] 1790658119.642011 punch from 0x0550d9 to 0x055a12 (0x93a bytes) 000451[1] 1790658119.675978 mapwrite 0x5f42c3 thru 0x5f6222 (0x1f60 bytes) 000452[0] 1790658119.840111 write 0x4d429a thru 0x4d4576 (0x2dd bytes) 000454[0] 1790658119.882680 mapread 0x4f66dc thru 0x4f6fb9 (0x8de bytes) 000456[0] 1790658119.939070 trunc from 0x5f6223 to 0x2104a2 (0x3e5d80 bytes) 000459[1] 1790658120.052472 trunc from 0x2104a3 to 0x71aa87 (0x50a5e5 bytes) 000461[1] 1790658120.188557 mapread 0x3070e2 thru 0x307a98 (0x9b7 bytes) 000463[0] 1790658120.239914 mapread 0x68418d thru 0x6857d4 (0x1648 bytes) 000464[0] 1790658120.289739 mapread 0x5c969a thru 0x5c9f81 (0x8e8 bytes) 000465[0] 1790658120.318833 mapread 0x0313c6 thru 0x03204d (0xc88 bytes) 000466[1] 1790658120.355369 mapwrite 0x5bd06d thru 0x5bd585 (0x519 bytes) 000467[1] 1790658120.441568 trunc from 0x71aa88 to 0x3cb026 (0x34fa61 bytes) 000472[1] 1790658120.572490 write 0x6328b7 thru 0x6330a6 (0x7f0 bytes) 000473[1] 1790658120.602625 read 0x277a51 thru 0x277e1e (0x3ce bytes) 000474[1] 1790658120.634039 read 0x5eeab7 thru 0x5eeade (0x28 bytes) 000478[1] 1790658120.704469 punch from 0x13057d to 0x131c23 (0x16a7 bytes) 000479[1] 1790658120.708968 write 0x65fb23 thru 0x6618e0 (0x1dbe bytes) 000482[1] 1790658120.750057 write 0x6e0280 thru 0x6e1413 (0x1194 bytes) 000484[0] 1790658120.757559 punch from 0x0a3686 to 0x0a46ff (0x107a bytes) 000485[1] 1790658120.782915 mapwrite 0x812581 thru 0x812e56 (0x8d6 bytes) 000486[0] 1790658120.873369 write 0x48ecd7 thru 0x48fa80 (0xdaa bytes) 000488[1] 1790658120.927554 trunc from 0x812e57 to 0x9147de (0x101988 bytes) 000491[0] 1790658121.026606 write 0x10498b thru 0x106802 (0x1e78 bytes) 000492[1] 1790658121.060134 write 0x129420 thru 0x129bc8 (0x7a9 bytes) 000496[1] 1790658121.197993 read 0x079e29 thru 0x07a7c2 (0x99a bytes) 000497[0] 1790658121.204590 trunc from 0x9147df to 0x3eb19c (0x529642 bytes) 000498[1] 1790658121.248128 read 0x31bfc5 thru 0x31d034 (0x1070 bytes) 000499[0] 1790658121.283461 write 0x900980 thru 0x90261d (0x1c9e bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:02:08 (1790658128) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:02:13 (1790658133) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.51752 s, 16.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.396752 s, 21.1 MB/s ldlm.namespaces.MGC192.168.202.140@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff919049be5000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff919061f51800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff919049be5000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff919049be5000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0408 s, 8.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.92164 s, 26.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.7308 s, 28.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.73083 s, 15.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.86891 s, 13.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.27518 s, 24.5 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:03:08 (1790658188) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304178 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238723 s, 4.4 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:03:17 (1790658197) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:03:46 (1790658226) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 01:04:15 (1790658255) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 01:04:22 (1790658262) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 01:04:30 (1790658270) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.60938 s, 16.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44705 s, 18.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64356 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05049 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79801 s, 9.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05652 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.34121 s, 11.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12561 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.27932 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8273 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.9414 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04427 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12716 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.84276 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.86519 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.70234 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.88281 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11316 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17974 s, 8.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:05:07 (1790658307) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Tue Sep 29 01:05:55 EDT 2026 + fsx 2 done: rc=0 Tue Sep 29 01:05:55 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Tue Sep 29 01:05:55 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Tue Sep 29 01:05:55 EDT 2026 + fsx 5 done: rc=0 Tue Sep 29 01:05:55 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:06:07 (1790658367) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 01:06:19 (1790658379) mmap test1: basic mmap operation (PASS, 0.10478s) mmap test2: MAP_PRIVATE not write back (PASS, 0.081102s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.2734s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7074s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15817s) mmap test6: check mmap write/read content on two nodes (PASS, 0.24168s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.5849s) mmap test11: multi-thread shared mmap (PASS, 1.6807s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1342s) PASS 18 (27s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:06:46 (1790658406) oleg240-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:06:51 (1790658411) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:06:58 (1790658418) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 01:07:05 (1790658425) now is 1790658428 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 new atime is 1790658489 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:08:16 (1790658496) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:08:24 (1790658504) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:08:32 (1790658512) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:08:40 (1790658520) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 01:08:44 (1790658524) utime: good mknod times 1790658525 <= 1790658526 <= 1790658526 for /mnt/lustre/f26a utime: good mknod times 1790658525 <= 1790658526 <= 1790658526 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (10s) == sanityn test 26b: sync mtime between ost and mds ====== 01:08:54 (1790658534) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:09:06 (1790658546) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-29 01:09:08.000000000 -0400 Birth: 2026-09-29 01:09:07.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-29 01:09:08.000000000 -0400 Birth: 2026-09-29 01:09:07.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:09:17 (1790658557) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:09:28 (1790658568) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:09:37 (1790658577) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.140@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 01:10:00 (1790658600) -rw-r--r-- 1 root root 0 Sep 29 01:10 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:10:10 (1790658610) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 43205 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:10:19 (1790658619) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 01:10:28 (1790658628) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 01:10:32 (1790658632) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32318 s, 39.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12567 s, 46.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93645 s, 27.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.12704 s, 24.6 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.23258 s, 23.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.57562 s, 20.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41623 s, 37.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4708 s, 35.6 MB/s PASS 32c (29s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:11:03 (1790658663) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:11:07 (1790658667) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:11:12 (1790658672) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:11:16 (1790658676) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 01:11:20 (1790658680) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid in IDLE state after 0 sec oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid in FULL state after 0 sec oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid in IDLE state after 0 sec oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid in FULL state after 0 sec oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff919049be5000.ost_server_uuid in IDLE state after 0 sec oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff919049be5000.ost_server_uuid in FULL state after 0 sec PASS 34 (143s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:13:43 (1790658823) fail_loc=0x80000317 Race attempt 0 Wait for 50510 50530 for 60 sec... - open/close 770 (time 1790658840.68 total 10.00 last 76.99) - open/close 1670 (time 1790658850.68 total 20.00 last 89.97) - open/close 2572 (time 1790658860.69 total 30.01 last 90.12) - open/close 3404 (time 1790658870.70 total 40.03 last 83.09) total: 4000 open/close in 47.06 seconds: 85.01 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.140@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.140@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:14:57 (1790658897) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47604 s, 35.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63315 s, 32.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85024 s, 28.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21845 s, 43.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4383 s, 36.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22451 s, 42.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41163 s, 37.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74844 s, 30.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39282 s, 37.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43255 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27044 s, 41.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7489536) after_dd(7439360) after(7490560) PASS 36 (324s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:20:21 (1790659221) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 - create 1529 (time 1790659235.97 total 10.00 last 152.88) - create 3111 (time 1790659245.98 total 20.01 last 158.14) - create 4699 (time 1790659255.98 total 30.01 last 158.77) - create 6385 (time 1790659265.98 total 40.01 last 168.53) - create 7943 (time 1790659276.00 total 50.03 last 155.53) - create 9054 (time 1790659286.05 total 60.08 last 110.56) total: 10000 create in 67.61 seconds: 147.92 ops/second PASS 37 (80s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:21:41 (1790659301) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:21:48 (1790659308) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:21:57 (1790659317) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 01:22:05 (1790659325) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:22:13 (1790659333) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (22s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:22:35 (1790659355) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:22:55 (1790659375) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:23:13 (1790659393) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:23:28 (1790659408) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:23:41 (1790659421) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:23:53 (1790659433) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (17s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:24:10 (1790659450) Conflict PASS 41d (19s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:24:29 (1790659469) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:24:44 (1790659484) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:24:56 (1790659496) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:25:10 (1790659510) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 01:25:24 (1790659524) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1069s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 01:43:14 (1790660594) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 01:43:25 (1790660605) open: File exists Conflict PASS 42b (11s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 01:43:36 (1790660616) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 01:43:49 (1790660629) Conflict PASS 42d (12s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 01:44:01 (1790660641) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (11s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 01:44:12 (1790660652) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 01:44:25 (1790660665) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 01:44:38 (1790660678) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 01:44:52 (1790660692) PASS 43a (59s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 01:45:50 (1790660750) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 01:46:04 (1790660764) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 01:46:18 (1790660778) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 01:46:31 (1790660791) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 01:46:45 (1790660805) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 01:46:57 (1790660817) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (11s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 01:47:08 (1790660828) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 01:47:22 (1790660842) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 01:47:26 (1790660846) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (122s) == sanityn test 43k: unlink vs create ==================== 01:49:28 (1790660968) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1110s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:07:58 (1790662078) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 29 02:08:06 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:08:11 (1790662091) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:08:25 (1790662105) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:08:42 (1790662122) Conflict PASS 44d (12s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:08:54 (1790662134) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:09:07 (1790662147) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:09:20 (1790662160) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:09:32 (1790662172) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:09:46 (1790662186) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:09:49 (1790662189) PASS 45a (106s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:11:35 (1790662295) Conflict PASS 45b (12s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:11:47 (1790662307) Conflict PASS 45c (12s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:12:00 (1790662320) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:12:13 (1790662333) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:12:27 (1790662347) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:12:40 (1790662360) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:12:53 (1790662373) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:13:06 (1790662386) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 02:13:09 (1790662389) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1114s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 02:31:43 (1790663503) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 02:31:56 (1790663516) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 02:32:09 (1790663529) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 02:32:21 (1790663541) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 02:32:34 (1790663554) Conflict PASS 46e (15s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 02:32:49 (1790663569) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 02:33:01 (1790663581) Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 02:33:13 (1790663593) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 02:33:26 (1790663606) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 02:33:30 (1790663610) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 02:33:33 (1790663613) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 02:33:37 (1790663617) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 02:33:40 (1790663620) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 02:33:43 (1790663623) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 02:33:47 (1790663627) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 02:33:50 (1790663630) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 02:33:54 (1790663634) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.139658 s, 73.3 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 02:34:06 (1790663646) 0+1 records in 0+1 records out 158 bytes copied, 0.0214385 s, 7.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (218834) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 02:34:16 (1790663656) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0131714 s, 77.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079349 s, 13.2 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 02:34:36 (1790663676) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.019995 s, 51.2 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 02:34:48 (1790663688) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580823 s, 18.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 02:34:55 (1790663695) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2449 0x991 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2449 0x991 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 02:35:03 (1790663703) llite.lustre-ffff919049be5000.xattr_cache=1 llite.lustre-ffff919061f51800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:04.000000000 -0400 Modify: 2026-09-29 02:35:04.000000000 -0400 Change: 2026-09-29 02:35:04.000000000 -0400 Birth: 2026-09-29 02:35:04.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:10.000000000 -0400 Modify: 2026-09-29 02:35:11.000000000 -0400 Change: 2026-09-29 02:35:11.000000000 -0400 Birth: 2026-09-29 02:35:10.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:16.000000000 -0400 Modify: 2026-09-29 02:35:16.000000000 -0400 Change: 2026-09-29 02:35:16.000000000 -0400 Birth: 2026-09-29 02:35:16.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:16.000000000 -0400 Modify: 2026-09-29 02:35:18.000000000 -0400 Change: 2026-09-29 02:35:18.000000000 -0400 Birth: 2026-09-29 02:35:16.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:23.000000000 -0400 Modify: 2026-09-29 02:35:24.000000000 -0400 Change: 2026-09-29 02:35:24.000000000 -0400 Birth: 2026-09-29 02:35:23.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 02:35:23.000000000 -0400 Modify: 2026-09-29 02:35:24.000000000 -0400 Change: 2026-09-29 02:35:24.000000000 -0400 Birth: 2026-09-29 02:35:23.000000000 -0400 PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 02:35:34 (1790663734) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 02:35:46 (1790663746) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 02:35:59 (1790663759) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 02:36:16 (1790663776) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 02:36:32 (1790663792) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 02:36:35 (1790663795) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 02:36:55 (1790663815) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 02:37:19 (1790663839) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790663840 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.649352 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (102s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 02:39:03 (1790663943) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 02:39:13 (1790663953) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 02:39:20 (1790663960) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 02:39:27 (1790663967) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00729022 s, 9.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 02:39:30 (1790663970) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00951539 s, 6.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 02:39:33 (1790663973) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 02:39:36 (1790663976) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.019734 s, 3.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 02:39:39 (1790663979) llite.lustre-ffff919049be5000.xattr_cache=1 llite.lustre-ffff919061f51800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 02:39:45 (1790663985) llite.lustre-ffff919049be5000.xattr_cache=1 llite.lustre-ffff919061f51800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 02:39:52 (1790663992) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0266953 s, 38.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.011628 s, 88.1 kB/s 231187: get lock1 231186: get lock2 231186: done 231186: get lock1 231187: done 231187: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231187: cannot get lock2: Resource deadlock avoided 231187: exit rc=0 231186: done 231186: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 02:40:03 (1790664003) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0942814 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359088 s, 29.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 231767 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 02:40:14 (1790664014) opening files: ........ closing files: ........ PASS 76 (71s) == sanityn test 77a: check FIFO NRS policy =============== 02:41:25 (1790664085) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455699 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425472 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116518 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158377 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987665 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203488 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300448 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285127 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288386 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383281 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412818 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404293 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274656 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420233 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469776 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490403 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.533285 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705738 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122918 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00763866 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581253 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122557 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119193 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133417 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976247 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157179 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153559 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141282 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217869 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067824 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170887 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755811 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995718 s, 10.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 02:41:34 (1790664094) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.651185 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167079 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156734 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217455 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170649 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176923 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116678 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208701 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338641 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355517 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324868 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439707 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347542 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347328 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489268 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344797 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47938 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998427 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114506 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078593 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111122 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463451 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134667 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138995 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164539 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110004 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139857 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116649 s, 9.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138135 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136012 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122107 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168289 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129974 s, 8.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.394136 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428308 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040908 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996481 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123306 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236389 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182381 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255014 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186692 s, 5.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233205 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301059 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263685 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361777 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301387 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291917 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256244 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468798 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892408 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538798 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10053 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118637 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145905 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245696 s, 427 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937055 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176411 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906819 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852327 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190819 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923082 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085565 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657349 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729507 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577273 s, 18.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (16s) == sanityn test 77c: check ORR NRS policy ================ 02:41:50 (1790664110) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.532989 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565575 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498168 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887138 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121762 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759589 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767232 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604483 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127155 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22801 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274166 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789368 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277608 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253522 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145375 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336373 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190404 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430809 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474187 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794428 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555482 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925653 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801883 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735995 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687012 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939453 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797092 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084856 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838964 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796073 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793706 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115293 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086141 s, 12.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.631972 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967165 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114232 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114922 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221917 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236646 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300493 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284479 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250067 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275931 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339147 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268095 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304476 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329982 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378733 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322223 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409877 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434637 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214659 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262795 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277375 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322784 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483754 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307297 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500288 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561998 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337391 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264171 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359323 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372562 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769716 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417234 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458975 s, 22.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.285851 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925938 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621483 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891688 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117901 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719812 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160029 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162081 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190644 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102631 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212988 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156572 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260255 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159586 s, 6.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225185 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262555 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338877 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468201 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617209 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204017 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346454 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324429 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297433 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342404 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466138 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375557 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487697 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670508 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369164 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125236 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713828 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723351 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336862 s, 311 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 02:42:05 (1790664125) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.376123 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074145 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889026 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171777 s, 6.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155483 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104374 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21325 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233046 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207429 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248278 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273298 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289358 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252407 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278308 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265586 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272705 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246505 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035695 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047187 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508135 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530973 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516301 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558988 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912186 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886136 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588542 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603525 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099823 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609628 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621274 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571879 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066756 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663215 s, 15.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309473 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559499 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774075 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083607 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174309 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217952 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220805 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21487 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228616 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286955 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176118 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22268 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234078 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256064 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205553 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265009 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35079 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774314 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324558 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457107 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612611 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458083 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326332 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511817 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649811 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674311 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448808 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617386 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070126 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708132 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893518 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870259 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908429 s, 11.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.564811 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059414 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710915 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170454 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199082 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223374 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242684 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296292 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308722 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269898 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337306 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382249 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342598 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167422 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444525 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391788 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459859 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520006 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854784 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039635 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060642 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959523 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489368 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663413 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696857 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241168 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251584 s, 417 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013307 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933549 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510557 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629125 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944905 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114031 s, 92.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (13s) == sanityn test 77e: check TBF NID nrs policy ============ 02:42:18 (1790664138) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.40@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230169 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503429 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612809 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826794 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106906 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119198 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117423 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137159 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14959 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176997 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185078 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201605 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205896 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218187 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208788 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217775 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252019 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926369 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162305 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988865 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127031 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143475 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1277 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15762 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123427 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164794 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13323 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144566 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192093 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173608 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1614 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189653 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207186 s, 5.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410848 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542374 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743452 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157029 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184048 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19317 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216682 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244662 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19221 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214666 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196009 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226326 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245303 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266267 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257215 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280877 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238178 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800108 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591164 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756341 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751088 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686464 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111626 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132123 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107242 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107144 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126962 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11234 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114364 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128299 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110265 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133194 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118768 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.406457 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624786 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105983 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118387 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118833 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145725 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15382 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238524 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237218 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220053 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214069 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313526 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310458 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31922 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293698 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288465 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396182 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483576 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046491 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05107 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314378 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320393 s, 327 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383388 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346378 s, 303 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408766 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339913 s, 308 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338408 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545804 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390626 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587515 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117536 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526749 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780404 s, 13.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44616 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862902 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966799 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787043 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140306 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124467 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515552 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154823 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118117 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167838 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33432 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312009 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279447 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23158 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330648 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322721 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375178 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029551 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590374 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382877 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253263 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397165 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218015 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275306 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037167 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038837 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417126 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428972 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044885 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450599 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474077 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489346 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504984 s, 20.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 02:42:40 (1790664160) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.292372 s, 57.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687033 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10347 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983528 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962333 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738782 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101399 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956601 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052999 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140676 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277823 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184561 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836613 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160444 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309744 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049315 s, 21.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233641 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531148 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536823 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917203 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501575 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646894 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958351 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103633 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734976 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486334 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653991 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561212 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588704 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105564 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949378 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920762 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103676 s, 10.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.704431 s, 23.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07614 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136235 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240493 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262737 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189642 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28744 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277249 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376426 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433323 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365772 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449935 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387776 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496927 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46634 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511616 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520218 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162646 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047805 s, 21.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301255 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143158 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179866 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130053 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173098 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140345 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22028 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02267 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177362 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195402 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203998 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168629 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334109 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055684 s, 18.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569205 s, 29.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519115 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478903 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978331 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121086 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146143 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172409 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115033 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11329 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111104 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215361 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156101 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212928 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249772 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252949 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257908 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106315 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525589 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532207 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396177 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545679 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601451 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503057 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643687 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399323 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406972 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512358 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990662 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535469 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669923 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621541 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776736 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664693 s, 15.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404754 s, 41.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741811 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794005 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939444 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116614 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120968 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711004 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140933 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144511 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160132 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218507 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254123 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154463 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191049 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19507 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220036 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195007 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154263 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781613 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107351 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051919 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037549 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166768 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184181 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197559 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198342 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271565 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261077 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259215 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322331 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336835 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357996 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353557 s, 29.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 02:43:05 (1790664185) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 02:43:16 (1790664196) oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 02:43:28 (1790664208) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 02:43:51 (1790664231) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519405 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691967 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919091 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12274 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211442 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243826 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256088 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252633 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295165 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283836 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279773 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274186 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280634 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326873 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265502 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312284 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267413 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411819 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193708 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834374 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269734 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685947 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064188 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531142 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658911 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355529 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486362 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586274 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758031 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052889 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680744 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734059 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10414 s, 10.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62798 s, 15.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5722 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (62s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 02:44:53 (1790664293) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33296 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960754 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148601 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150784 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133045 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110774 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687269 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225954 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150665 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183184 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197419 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805947 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241234 s, 4.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174591 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15579 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238626 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155881 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522706 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433031 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071152 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460513 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835103 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837388 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377266 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039206 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397988 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716356 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424606 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484371 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705896 s, 14.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675847 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681976 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740919 s, 14.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.953 s, 5.3 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9579 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28644 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622165 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131634 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138736 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11082 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141889 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123414 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154368 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193789 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207512 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213278 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219899 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244235 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232667 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272473 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262679 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30085 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589264 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714468 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653762 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067929 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696838 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072127 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815399 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081978 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912187 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978205 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100238 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986334 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107068 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110785 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119266 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117533 s, 8.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8923 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9199 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 02:47:26 (1790664446) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30643 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115673 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11291 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115916 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120616 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148009 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175016 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204908 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197439 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195064 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195843 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816322 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21251 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252199 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197459 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289374 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333007 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072632 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464942 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886053 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852964 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572358 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393492 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660415 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675003 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895106 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787223 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692185 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756393 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599835 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801907 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924558 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908216 s, 11.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0159 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9993 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32827 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107298 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12144 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109008 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191007 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231378 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209815 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236053 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171164 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186394 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221015 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230296 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319369 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360798 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402698 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433617 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38802 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653752 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636629 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520806 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955937 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101152 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114531 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923497 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777042 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10579 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100657 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117957 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11266 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112933 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116979 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164805 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127301 s, 8.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9319 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0002 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (150s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 02:49:56 (1790664596) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.491543 s, 34.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533789 s, 19.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517172 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285796 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100689 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123092 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119083 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131923 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150503 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151478 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167192 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175156 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180211 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181207 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17599 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171861 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186038 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597546 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575524 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295328 s, 35.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584912 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540831 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602868 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819966 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383329 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791528 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452986 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446111 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487288 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054213 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548099 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807267 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103725 s, 10.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.61516 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9902 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.40@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.40@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.707039 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991135 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742964 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111793 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127332 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111116 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730733 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243766 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23102 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146363 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259301 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160929 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250418 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217574 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281387 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323886 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258742 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192352 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133349 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200129 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197375 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199293 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211056 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194186 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202226 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214595 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204681 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14477 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218028 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152305 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214247 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155673 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168024 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=11 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.23241 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0647 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.40@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663733 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770926 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724976 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119165 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16113 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182238 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243973 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283995 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283064 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248779 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295841 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307443 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34963 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317199 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324059 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353156 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398438 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00932723 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735236 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074062 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421486 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210102 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471924 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043734 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230181 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527527 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552887 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355189 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579422 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376582 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689853 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045541 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051444 s, 20.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.77559 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2562 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571919 s, 29.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483509 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926583 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766182 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124914 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209607 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155017 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150956 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170386 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180292 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234578 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236617 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292528 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302577 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366683 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372205 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380484 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749744 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531055 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262259 s, 400 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683431 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100809 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115521 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109907 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120185 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068669 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643674 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775869 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261362 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235973 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299001 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384496 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337676 s, 31.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8128 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90274 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.29358 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95829 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.937303 s, 17.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758633 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100871 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122681 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101753 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100472 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143779 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978739 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127295 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275037 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263965 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209782 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339006 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249067 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364363 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35546 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371515 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531489 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719245 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884012 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739024 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773269 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556426 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630575 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676366 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666067 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776408 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729191 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835344 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112657 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856992 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118732 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103235 s, 10.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7948 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6229 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401019 s, 41.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539724 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621215 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100325 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913059 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122073 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193483 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234011 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159429 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251068 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277648 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208156 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284525 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256247 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222116 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216251 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336358 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431708 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513834 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662619 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544594 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419546 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444016 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448469 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749433 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079986 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650734 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065854 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649077 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065326 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732612 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080403 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826014 s, 12.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.8981 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0432 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.645538 s, 26.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624631 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074304 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223499 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225488 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233536 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236461 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292238 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271465 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309943 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302911 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301495 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320304 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307549 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310908 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367879 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377807 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193346 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00938968 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474089 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100208 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00582113 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957411 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174077 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418018 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235293 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00300263 s, 349 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00202445 s, 518 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265517 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388007 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261074 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175472 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017514 s, 59.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.777 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93826 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.661 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96886 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (423s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 02:56:59 (1790665019) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.860811 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11929 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161975 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148507 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348946 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311054 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312514 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29754 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192234 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.594914 s, 1.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559326 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520335 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.620936 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.597942 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468768 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344985 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.585727 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176409 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181175 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13694 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104706 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830231 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698431 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177666 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216202 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146334 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150372 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132303 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148709 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141295 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220721 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162991 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265082 s, 4.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5097 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90012 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (59s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 02:57:58 (1790665078) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.40@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.40@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498132 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839267 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883383 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980304 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201388 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247173 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260658 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231679 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250125 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242408 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276661 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270469 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276992 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303522 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34411 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361128 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309145 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff919061f51800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161694 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159116 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159609 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164182 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203852 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168278 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178915 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178074 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178391 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182484 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183539 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243222 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196771 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232017 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201807 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208119 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=3 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.42334 s, 16.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91822 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (49s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 02:58:47 (1790665127) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.40@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.625537 s, 26.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046865 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073663 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834904 s, 12.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120642 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152161 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076602 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311923 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203525 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292308 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235821 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215482 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37816 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353694 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310146 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346988 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306897 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003421 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422712 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639681 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717566 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110774 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772301 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702253 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606844 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479667 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707818 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137873 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394782 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425146 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222732 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215104 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029067 s, 36.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.1268 s, 14.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43031 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (49s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 02:59:36 (1790665176) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.75337 s, 22.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369017 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970128 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881658 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160674 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192963 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287542 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235513 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300095 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295734 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312867 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256917 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253807 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250514 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263625 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333996 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323544 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295384 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316668 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583777 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345884 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252223 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368605 s, 28.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760337 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339654 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385418 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588883 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551314 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453106 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778647 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540024 s, 19.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706722 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922987 s, 11.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9773 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92368 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.81038 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96597 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (95s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:01:11 (1790665271) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09242 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869682 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439849 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140712 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1465 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12703 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171969 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142212 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909993 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203134 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255732 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335589 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242374 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329444 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237989 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329544 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295848 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520933 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346134 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201825 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217742 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978296 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506472 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226851 s, 462 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745517 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660922 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536611 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556856 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508984 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687516 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209838 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196102 s, 53.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9045 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0069 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:02:27 (1790665347) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467549 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690071 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775223 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899107 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181304 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16055 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248595 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257614 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269798 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282382 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306102 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3162 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304967 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307866 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304122 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339042 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349773 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919061f51800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445665 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183741 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259191 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430631 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478106 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512714 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226512 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347956 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539125 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602285 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376174 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612882 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474596 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497226 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605583 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357256 s, 29.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.34705 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99131 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg240-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.634991 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681526 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056229 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163889 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160432 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234411 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264638 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179008 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167204 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2508 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216706 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324014 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202288 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221177 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269819 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291908 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479924 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227345 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174682 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275021 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289458 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026743 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266155 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409085 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308967 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319789 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365446 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375473 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679125 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392856 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423312 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433446 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454155 s, 23.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6495 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84005 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7745 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96495 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (139s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:04:46 (1790665486) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg240-client.virtnet /mnt/lustre (opts:) Stopping client oleg240-client.virtnet /mnt/lustre opts: Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg240-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg240-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg240-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.40,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg240-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.40,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg240-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.40,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71018/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.40,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg240-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: kmemleak disabled oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre Starting client oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre Started clients oleg240-client.virtnet: 192.168.202.140@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Started clients oleg240-client.virtnet: 192.168.202.140@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff919045d18000.idle_timeout=debug osc.lustre-OST0000-osc-ffff91904ac97800.idle_timeout=debug osc.lustre-OST0001-osc-ffff919045d18000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91904ac97800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404966 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069846 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821115 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128709 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181514 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164162 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15967 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199976 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210489 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177802 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206088 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193076 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20568 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199821 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206665 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221815 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215352 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159453 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412316 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187466 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403477 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230668 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287612 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052476 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047845 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766611 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584485 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668072 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436336 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448434 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605815 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512166 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751668 s, 13.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.83188 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20766 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479803 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326339 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655976 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814833 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649204 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161537 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958422 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172734 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190723 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198678 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210358 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212184 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227603 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243707 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160945 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180941 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269003 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339488 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366199 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358306 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037756 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404758 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519217 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641425 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607842 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635712 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637275 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888874 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825827 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130434 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929285 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960724 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109624 s, 9.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7016 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89774 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (216s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 03:08:22 (1790665702) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (20s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 03:08:42 (1790665722) On MGS 192.168.202.140, active = nodemap.active=1 On MGS 192.168.202.140, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.547337 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081735 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768483 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183168 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224515 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260884 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240537 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248794 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260936 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248084 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286599 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304363 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306718 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349047 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363634 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372034 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322817 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff919045d18000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638746 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602783 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667837 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366144 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609109 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800373 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774419 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588578 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519082 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809049 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537097 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636495 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466835 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549879 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784025 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559705 s, 18.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.19951 s, 14.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13637 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.140, TBF.id = On MGS 192.168.202.140, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.521523 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635268 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047592 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915863 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164051 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188885 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187683 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216964 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135489 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872496 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306076 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166018 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161752 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216682 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23476 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919045d18000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff919045d18000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254364 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048609 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218401 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687607 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310747 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425402 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319442 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442017 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601871 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473644 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504711 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268404 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398406 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313698 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430543 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712177 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381361 s, 27.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98908 s, 21.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.74549 s, 60.1 MB/s Read runtime is 2 s, speed is 50.000000 IOPS read rate (50.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg240-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 On MGS 192.168.202.140, active = nodemap.active=1 On MGS 192.168.202.140, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.370558 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388304 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065836 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552871 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790143 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636031 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134319 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138133 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105699 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121463 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148536 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161233 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114434 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196784 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20573 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206901 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203244 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00234773 s, 447 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00235533 s, 445 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101722 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889101 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431157 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025955 s, 404 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08445 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392868 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612125 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562662 s, 18.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303866 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240976 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414508 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183741 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233944 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232307 s, 45.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.28754 s, 16.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88402 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.140, TBF.id = On MGS 192.168.202.140, active = nodemap.active=0 PASS 77kj (166s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:11:28 (1790665888) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (8s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:11:36 (1790665896) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg240-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.09207 s, 256 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg240-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 39.9602 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:12:34 (1790665954) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.420004 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157816 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153184 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17478 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206518 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198447 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270595 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149192 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253765 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19591 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249213 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37153 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126585 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26653 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261093 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230982 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424226 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919045d18000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196101 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334235 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060368 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711949 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701602 s, 14.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505068 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360775 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621465 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447733 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443067 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063785 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529486 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163946 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536676 s, 19.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596828 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694627 s, 15.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg240-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg240-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.87231 s, 15.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17051 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527437 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091437 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726106 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982646 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182776 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209861 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232787 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247276 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243005 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303824 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286721 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336953 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329307 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356192 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295108 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380088 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403477 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919045d18000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff919045d18000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433138 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907802 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476933 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605928 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527325 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564484 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575479 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555515 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125706 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960346 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654335 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662001 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823026 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739215 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475062 s, 22.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff919045d18000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126957 s, 8.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg240-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg240-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.23515 s, 14.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36675 s, 19.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 03:14:03 (1790666043) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (10s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:14:13 (1790666053) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (109s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 03:16:03 (1790666163) mds.MDS.mdt.nrs_policies=tbf oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg240-client: oleg240-server: ssh exited with exit code 36 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 oleg240-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg240-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg240-client: oleg240-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 03:16:39 (1790666199) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273441 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 03:17:35 (1790666255) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (63s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 03:18:38 (1790666318) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 03:18:48 (1790666328) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 03:18:49.000000000 -0400 Modify: 2026-09-29 03:18:49.000000000 -0400 Change: 2026-09-29 03:18:50.000000000 -0400 Birth: 2026-09-29 03:18:49.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 03:19:05 (1790666345) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 03:19:08 (1790666348) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 03:19:11 (1790666351) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 03:19:14 (1790666354) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 03:19:17 (1790666357) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 03:19:20 (1790666360) - open/close 864 (time 1790666375.04 total 10.00 last 86.39) - open/close 1664 (time 1790666385.05 total 20.01 last 79.92) total: 2500 open/close in 29.09 seconds: 85.94 ops/second snapshot_time 1790666474.363631825 secs.nsecs start_time 1790666362.097811517 secs.nsecs elapsed_time 112.265820308 secs.nsecs open 2505 samples [usecs] 131 29688 3451616 8996791716 close 2504 samples [usecs] 142 5058 857897 533103833 mknod 2501 samples [usecs] 531 29551 2993272 6711142734 unlink 2500 samples [usecs] 640 88284 4890271 46427879721 mkdir 2 samples [usecs] 1239 2302 3541 6834325 rmdir 2 samples [usecs] 10271 10394 20665 213528677 rename 2500 samples [usecs] 450 48556 16042032 222176275438 getattr 3763 samples [usecs] 7 11535 210153 360647507 parallel_rename_file 2500 samples [usecs] 450 48556 16042032 222176275438 crossdir_rename 2500 samples [usecs] 450 48556 16042032 222176275438 readpage 24 samples [usecs] 3513 58990 494293 15230431559 PASS 81d (120s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 03:21:20 (1790666480) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 03:21:28 (1790666488) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 03:21:31 (1790666491) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 03:21:44 (1790666504) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790666506 Finishing test test1 at 1790666510 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 03:21:55 (1790666515) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 03:21:59 (1790666519) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 03:22:02 (1790666522) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 03:22:05 (1790666525) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4790 0x12b6 0x240000400 1 4753 0x1291 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4791 0x12b7 0x240000400 1 4754 0x1292 0x280000400 rc1=0 and rc2=0 PASS 93 (16s) == sanityn test 94: signal vs CP callback race =========== 03:22:21 (1790666541) osc.lustre-OST0000-osc-ffff919045d18000.idle_timeout=0 osc.lustre-OST0000-osc-ffff91904ac97800.idle_timeout=0 osc.lustre-OST0001-osc-ffff919045d18000.idle_timeout=0 osc.lustre-OST0001-osc-ffff91904ac97800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.329702 s, 24.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.01032 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 290460 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0845312 s, 48.5 kB/s osc.lustre-OST0000-osc-ffff919045d18000.idle_timeout=debug osc.lustre-OST0000-osc-ffff91904ac97800.idle_timeout=debug osc.lustre-OST0001-osc-ffff919045d18000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91904ac97800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 03:22:37 (1790666557) llite.lustre-ffff919045d18000.fast_read=0 llite.lustre-ffff91904ac97800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0632373 s, 33.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff919045d18000.fast_read=1 llite.lustre-ffff91904ac97800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 03:22:57 (1790666577) llite.lustre-ffff919045d18000.fast_read=0 llite.lustre-ffff91904ac97800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00373807 s, 3.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff919045d18000.fast_read=1 llite.lustre-ffff91904ac97800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 03:23:10 (1790666590) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 03:23:13 (1790666593) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00397195 s, 1.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 03:23:20 (1790666600) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174615 s, 12.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 03:23:28 (1790666608) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198185 s, 10.6 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 03:23:35 (1790666615) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00899154 s, 1.3 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 03:23:42 (1790666622) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150284 s, 273 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 03:23:50 (1790666630) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00940256 s, 436 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 03:23:57 (1790666637) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137775 s, 297 kB/s PASS 101c (10s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 03:24:07 (1790666647) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (9s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 03:24:16 (1790666656) 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.202.140@tcp:/lustre /mnt/lustre Starting client: oleg240-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.202.140@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.202.140@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 PASS 102b (18s) == sanityn test 103: Test size correctness with lockahead ========================================================== 03:24:34 (1790666674) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790666677 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790666681 Finishing test test23 at 1790666681 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 03:24:48 (1790666688) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 03:24:52 (1790666692) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 03:25:10 (1790666710) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 03:25:14 (1790666714) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0575347 s, 17.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 03:25:17.000000000 -0400 Modify: 2026-09-29 03:25:17.000000000 -0400 Change: 2026-09-29 03:25:17.000000000 -0400 Birth: 2026-09-29 03:25:16.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1790666716 1790666717 1790666717 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 03:25:17.000000000 -0400 Modify: 2026-09-29 03:25:17.000000000 -0400 Change: 2026-09-29 03:25:17.000000000 -0400 Birth: 2026-09-29 03:25:16.000000000 -0400 1790666717 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 03:25:23 (1790666723) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 03:25:30 (1790666730) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363363 s, 28.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4760 0x1298 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 03:25:39 (1790666739) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.435015 s, 24.1 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4801 0x12c1 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 03:25:51 (1790666751) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0116282 s, 860 kB/s llite.lustre-ffff9190451cd000.hybrid_io=0 llite.lustre-ffff919049293000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.77481 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 03:26:03 (1790666763) 192.168.202.140@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.140@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: kmemleak disabled oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.140@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: kmemleak disabled oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 PASS 109 (71s) == sanityn test 110: do not grant another lock on resend ========================================================== 03:27:14 (1790666834) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 03:27:16.000000000 -0400 Modify: 2026-09-29 03:27:16.000000000 -0400 Change: 2026-09-29 03:27:16.000000000 -0400 Birth: 2026-09-29 03:27:16.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg240-client.virtnet oleg240-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 03:30:05 (1790667005) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 03:30:09 (1790667009) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 03:30:13 (1790667013) lustre-OST0000-osc-ffff9190457e7000 active. lustre-OST0001-osc-ffff9190457e7000 active. lustre-OST0000-osc-ffff9190457e7000 active. lustre-OST0001-osc-ffff9190457e7000 active. MGC192.168.202.140@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 03:30:20 (1790667020) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 03:30:22 (1790667022) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 03:30:25 (1790667025) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 03:30:28 (1790667028) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0118386 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0725556 s, 282 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 03:30:37 (1790667037) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 03:30:47 (1790667047) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1884 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2685 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1988 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2537 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1968 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2347 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2258 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1147 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1262 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.285 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2313 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3083 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1887 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2258 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2871 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3724 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3926 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3773 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91904505f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91904505f800.ost_server_uuid in IDLE state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 9572 sec ============== 03:32:13 (1790667133) === sanityn: start cleanup 03:32:15 (1790667135) === Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg240-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 03:35:53 (1790667353) ===