-----============= acceptance-small: sanityn ============----- Tue Sep 8 08:17:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg255-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 08:18:05 (1788869885) === Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Mount client oleg255-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg255-client.virtnet: executing check_config_client /mnt/lustre oleg255-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg255-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf40343b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf405439800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf40343b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf405439800.idle_timeout=debug disable quota as required oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 08:18:41 (1788869921) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589199 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7647] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:18:43 (1788869923) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:18:51 (1788869931) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:18:57 (1788869937) -rw-r--r-- 1 root root 0 Sep 8 08:18 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:19:04 (1788869944) -rw-r--r-- 1 root root 0 Sep 8 08:19 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:19:11 (1788869951) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:19:18 (1788869958) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:19:25 (1788869965) total 1 -rwxrwxrwx 1 root root 0 Sep 8 08:18 f2a -rwxrwxrwx 1 root root 0 Sep 8 08:19 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 8 08:18 f2a -rwxrwxrwx 1 root root 0 Sep 8 08:19 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00481691 s, 106 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:19:32 (1788869972) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 08:19:37 (1788869977) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.400765 s, 5.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:19:46 (1788869986) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:19:53 (1788869993) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:20:01 (1788870001) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 08:20:08 (1788870008) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:20:15 (1788870015) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:20:21 (1788870021) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:20:28 (1788870028) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 08:20:36 (1788870036) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045446 s, 23.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0365408 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:20:44 (1788870044) 1+0 records in 1+0 records out 1 byte copied, 0.0120727 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0445071 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145479 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0124939 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0162517 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0218368 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140519 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151232 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0236255 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.022681 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0125465 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0419653 s, 0.0 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:20:53 (1788870053) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00971111 s, 316 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0324417 s, 126 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000280684 s, 10.9 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:21:00 (1788870060) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:21:08 (1788870068) warning: '-runtime' deprecated, use '-t runtime' instead - link 283 (time 1788870080.71 total 10.02 last 28.26) - link 566 (time 1788870090.75 total 20.05 last 28.20) - link 863 (time 1788870100.77 total 30.07 last 29.65) - link 1132 (time 1788870110.78 total 40.08 last 26.88) - link 1417 (time 1788870120.78 total 50.08 last 28.48) - link 1752 (time 1788870130.79 total 60.09 last 33.49) - link 2063 (time 1788870140.82 total 70.12 last 30.99) - link 2307 (time 1788870150.83 total 80.13 last 24.38) - link 2624 (time 1788870160.85 total 90.15 last 31.65) total: 2938 link in 100.03 seconds: 29.37 ops/second using seed 3154447550 running for 100 seconds - stat 4060 (time 1788870082 ; total 11 ; last 11) - stat 8377 (time 1788870093 ; total 22 ; last 11) - stat 10000 (time 1788870096 ; total 25 ; last 3) - stat 14868 (time 1788870107 ; total 36 ; last 11) - stat 20000 (time 1788870116 ; total 45 ; last 9) - stat 23931 (time 1788870127 ; total 56 ; last 11) - stat 26973 (time 1788870138 ; total 67 ; last 11) - stat 28951 (time 1788870149 ; total 78 ; last 11) - stat 30000 (time 1788870153 ; total 82 ; last 4) - stat 32861 (time 1788870164 ; total 93 ; last 11) total: 34969 stats in 100 seconds: 349.690002 stats/second - unlinked 0 (time 1788870121 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2938) error: No such file or directory total: 2937 unlinks in 79 seconds: 37.177216 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21445) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21448) - No such process PASS 12 (139s) == sanityn test 13: test directory page revocation ======= 08:23:27 (1788870207) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 08:23:36 (1788870216) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22629 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 08:23:44 (1788870224) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23210 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 08:23:50 (1788870230) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23788 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 08:23:58 (1788870238) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24370 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 08:24:05 (1788870245) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24952 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 08:24:16 (1788870256) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (54s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 08:25:10 (1788870310) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.520926 s, 16.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.6147 s, 13.6 MB/s Chance of close/open is 1 in 50 Seed set to 315 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read 000020[0] 1788870316.969867 mapwrite 0x450b45 thru 0x45af91 (0xa44d bytes) 000021[1] 1788870317.343764 mapwrite 0x54690a thru 0x553c38 (0xd32f bytes) 000025[0] 1788870317.609470 punch from 0x51d7eb to 0x520608 (0x2e1e bytes) 000027[1] 1788870317.717031 write 0x4a37ff thru 0x4ae0aa (0xa8ac bytes) 000029[1] 1788870317.803314 mapread 0x50150d thru 0x50b127 (0x9c1b bytes) 000031[1] 1788870317.877516 punch from 0x49e185 to 0x4a3a1a (0x5896 bytes) truncating to largest ever: 0xb7243 000032[1] 1788870317.900925 trunc from 0x553c39 to 0x0b7242 (0x49c9f6 bytes) truncating to largest ever: 0x5e1e94 000033[0] 1788870317.926221 trunc from 0x0b7243 to 0x5e1e93 (0x52ac51 bytes) 000034[1] 1788870318.006529 trunc from 0x5e1e94 to 0x03d4ee (0x5a49a5 bytes) 000036[1] 1788870318.124708 write 0x96f161 thru 0x971787 (0x2627 bytes) truncating to largest ever: 0x73bf4c 000037[0] 1788870318.148102 trunc from 0x971788 to 0x73bf4b (0x23583c bytes) 000038[0] 1788870318.181203 mapwrite 0x480827 thru 0x4896e7 (0x8ec1 bytes) 000042[1] 1788870318.415406 punch from 0x0853dd to 0x08b8b4 (0x64d8 bytes) 000043[1] 1788870318.459990 punch from 0x69fea1 to 0x6a50eb (0x524b bytes) 000044[1] 1788870318.558190 trunc from 0x73bf4c to 0x4fb45d (0x240aee bytes) 000046[1] 1788870318.654697 mapwrite 0x41c244 thru 0x4283dd (0xc19a bytes) 000047[1] 1788870318.854411 punch from 0x3f11f8 to 0x3f35e6 (0x23ef bytes) 000048[0] 1788870318.932260 read 0x4dca8c thru 0x4de347 (0x18bc bytes) 000049[0] 1788870319.008053 mapread 0x489c85 thru 0x497bd4 (0xdf50 bytes) 000051[1] 1788870319.143850 write 0x931887 thru 0x932ecf (0x1649 bytes) 000052[1] 1788870319.181895 punch from 0x8510a3 to 0x8521e3 (0x1141 bytes) 000055[1] 1788870319.234566 trunc from 0x932ed0 to 0x6d0135 (0x262d9a bytes) truncating to largest ever: 0x8d80cd 000057[1] 1788870319.296649 trunc from 0x6d0136 to 0x8d80cc (0x207f97 bytes) 000058[1] 1788870319.317376 mapread 0x1cb491 thru 0x1cf019 (0x3b89 bytes) 000059[0] 1788870319.365037 trunc from 0x8d80cd to 0x8d0054 (0x8078 bytes) 000060[1] 1788870319.450114 punch from 0x03e324 to 0x040a5e (0x273b bytes) 000063[1] 1788870319.570106 trunc from 0x8d0055 to 0x59eea1 (0x3311b3 bytes) 000064[0] 1788870319.694163 mapread 0x52d064 thru 0x53aafe (0xda9b bytes) 000066[0] 1788870319.830794 mapwrite 0x0c26ef thru 0x0c9a4f (0x7361 bytes) 000067[1] 1788870320.184495 mapread 0x08dfd5 thru 0x0982c1 (0xa2ed bytes) 000068[0] 1788870320.318674 punch from 0x36e886 to 0x37944a (0xabc5 bytes) 000069[1] 1788870320.409658 read 0x069fca thru 0x06da8f (0x3ac6 bytes) truncating to largest ever: 0x8f2a59 000071[1] 1788870320.458213 trunc from 0x59eea2 to 0x8f2a58 (0x353bb7 bytes) 000072[0] 1788870320.556997 mapread 0x5a069d thru 0x5ae81d (0xe181 bytes) 000074[1] 1788870320.690267 punch from 0x0c93d9 to 0x0cb308 (0x1f30 bytes) 000075[0] 1788870320.731822 punch from 0x7afda8 to 0x7bc88b (0xcae4 bytes) 000076[1] 1788870320.846081 mapread 0x23c586 thru 0x24995b (0xd3d6 bytes) 000077[0] 1788870320.892480 mapwrite 0x9e7e98 thru 0x9f2fcb (0xb134 bytes) 000078[0] 1788870321.093166 mapwrite 0x8db208 thru 0x8e8279 (0xd072 bytes) 000080[0] 1788870321.236238 trunc from 0x9f2fcc to 0x8a580d (0x14d7be bytes) 000081[1] 1788870321.258754 read 0x5bd815 thru 0x5c61fa (0x89e6 bytes) 000083[0] 1788870321.380159 punch from 0x4b7a17 to 0x4b9ad3 (0x20bd bytes) 000085[1] 1788870321.508903 mapwrite 0x799825 thru 0x7a72e9 (0xdac5 bytes) 000086[0] 1788870321.685900 read 0x1fdad7 thru 0x207735 (0x9c5f bytes) 000090[0] 1788870321.781710 read 0x028f8d thru 0x035b94 (0xcc08 bytes) 000091[1] 1788870321.828765 read 0x6596dc thru 0x667174 (0xda99 bytes) 000094[1] 1788870321.922311 punch from 0x78f492 to 0x78f60a (0x179 bytes) 000095[0] 1788870321.958871 punch from 0x7bc92f to 0x7bf211 (0x28e3 bytes) 000097[0] 1788870322.047150 mapread 0x24dc3d thru 0x24f12f (0x14f3 bytes) 000098[1] 1788870322.186268 mapread 0x2b0671 thru 0x2b1295 (0xc25 bytes) 000099[1] 1788870322.266971 mapread 0x63d0c2 thru 0x640b30 (0x3a6f bytes) 000100[1] 1788870322.362455 punch from 0x22c986 to 0x23889e (0xbf19 bytes) 000101[1] 1788870322.416042 punch from 0x79022e to 0x7929af (0x2782 bytes) 000104[1] 1788870322.523591 mapwrite 0x31a654 thru 0x32591e (0xb2cb bytes) 000105[0] 1788870322.732565 mapread 0x6c32ad thru 0x6d27c9 (0xf51d bytes) 000107[0] 1788870322.935572 mapread 0x50cfad thru 0x51629a (0x92ee bytes) 000108[0] 1788870323.006521 mapwrite 0x5d8597 thru 0x5dd777 (0x51e1 bytes) 000114[0] 1788870323.385261 mapread 0x5da494 thru 0x5e36fa (0x9267 bytes) 000115[0] 1788870323.454965 mapwrite 0x312d3a thru 0x31e602 (0xb8c9 bytes) 000117[1] 1788870323.696281 write 0x70907d thru 0x7169b4 (0xd938 bytes) 000119[1] 1788870323.819437 punch from 0x7162ca to 0x71cac9 (0x6800 bytes) 000121[0] 1788870323.866069 punch from 0x2fd834 to 0x3047f4 (0x6fc1 bytes) 000122[0] 1788870323.896257 mapread 0x3f33fc thru 0x3fb705 (0x830a bytes) 000123[0] 1788870323.937925 punch from 0x20c2d9 to 0x2157e8 (0x9510 bytes) 000124[1] 1788870323.966811 mapwrite 0x31c69c thru 0x325575 (0x8eda bytes) 000126[0] 1788870324.189485 read 0x89e062 thru 0x8a5567 (0x7506 bytes) 000127[1] 1788870324.228193 trunc from 0x8a580e to 0x5fe103 (0x2a770a bytes) 000129[0] 1788870324.300323 mapwrite 0x4886bb thru 0x491056 (0x899c bytes) 000130[0] 1788870324.632060 trunc from 0x5fe104 to 0x6f1830 (0xf372d bytes) 000132[0] 1788870324.674553 mapread 0x3ba659 thru 0x3bec35 (0x45dd bytes) 000133[1] 1788870324.767560 write 0x9cdd36 thru 0x9d312e (0x53f9 bytes) 000134[0] 1788870324.874760 mapread 0x7515fa thru 0x76094e (0xf355 bytes) 000135[0] 1788870325.297365 mapwrite 0x2f2bf8 thru 0x2f5e8a (0x3293 bytes) 000137[0] 1788870325.457948 write 0x834ce3 thru 0x83545b (0x779 bytes) 000138[1] 1788870325.481438 punch from 0x21fab9 to 0x22e2de (0xe826 bytes) 000140[1] 1788870325.600059 mapwrite 0x407aef thru 0x40e2b5 (0x67c7 bytes) 000141[0] 1788870325.831232 mapread 0x881877 thru 0x88fd96 (0xe520 bytes) 000142[1] 1788870325.900023 mapread 0x9c3ed5 thru 0x9cd829 (0x9955 bytes) 000143[1] 1788870325.934718 punch from 0x3f8466 to 0x3ff756 (0x72f1 bytes) 000144[0] 1788870325.962332 write 0x86e8a6 thru 0x877129 (0x8884 bytes) truncating to largest ever: 0x9da206 000145[0] 1788870326.001249 trunc from 0x9d312f to 0x9da205 (0x70d7 bytes) 000148[1] 1788870326.112896 read 0x98241b thru 0x982c69 (0x84f bytes) 000150[0] 1788870326.162643 mapread 0x8ad68d thru 0x8b7647 (0x9fbb bytes) 000151[0] 1788870326.188060 mapwrite 0x62eac7 thru 0x631aee (0x3028 bytes) 000153[0] 1788870326.277351 punch from 0x41cc76 to 0x41ea1c (0x1da7 bytes) 000155[1] 1788870326.302872 trunc from 0x9da206 to 0x55e9c3 (0x47b842 bytes) 000157[1] 1788870326.378221 punch from 0x4ff5fd to 0x5067e9 (0x71ed bytes) 000158[0] 1788870326.402245 read 0x193f5a thru 0x1966ff (0x27a6 bytes) 000160[0] 1788870326.424238 trunc from 0x55e9c4 to 0x5bba71 (0x5d0ae bytes) 000163[0] 1788870326.497430 mapread 0x4b54d8 thru 0x4c4fa0 (0xfac9 bytes) 000165[1] 1788870326.564424 trunc from 0x5bba72 to 0x5f20c7 (0x36656 bytes) 000170[0] 1788870326.691928 read 0x33be5c thru 0x33f968 (0x3b0d bytes) 000171[0] 1788870326.737399 read 0x5a0c3b thru 0x5a5a3c (0x4e02 bytes) 000172[0] 1788870326.758552 punch from 0x4ee0fe to 0x4f14bd (0x33c0 bytes) 000175[1] 1788870326.822273 read 0x1e735d thru 0x1effb0 (0x8c54 bytes) 000176[1] 1788870326.849486 mapwrite 0x0e105f thru 0x0ead25 (0x9cc7 bytes) 000177[1] 1788870326.980399 trunc from 0x5f20c8 to 0x33a796 (0x2b7931 bytes) 000178[1] 1788870327.054322 trunc from 0x33a797 to 0x45a30c (0x11fb76 bytes) 000181[0] 1788870327.094980 read 0x2bfb6f thru 0x2c2fde (0x3470 bytes) 000182[1] 1788870327.231309 mapread 0x0a8eda thru 0x0a97a7 (0x8ce bytes) 000183[1] 1788870327.263074 mapwrite 0x332e2f thru 0x337077 (0x4249 bytes) 000184[1] 1788870327.381958 mapwrite 0x1e2e98 thru 0x1f22f4 (0xf45d bytes) 000185[1] 1788870327.723530 trunc from 0x45a30d to 0x05ca16 (0x3fd8f6 bytes) 000186[1] 1788870327.802722 punch from 0x049da4 to 0x059b9d (0xfdfa bytes) 000187[1] 1788870327.814256 read 0x03e987 thru 0x041833 (0x2ead bytes) 000188[1] 1788870327.917096 trunc from 0x05ca17 to 0x964767 (0x907d51 bytes) 000190[1] 1788870327.970926 punch from 0x1e2139 to 0x1e3d24 (0x1bec bytes) 000191[0] 1788870328.048281 punch from 0x3b3709 to 0x3c128f (0xdb87 bytes) 000193[0] 1788870328.157293 punch from 0x8542c7 to 0x85dcfe (0x9a38 bytes) 000195[1] 1788870328.236204 mapread 0x0a740d thru 0x0b64a5 (0xf099 bytes) 000196[0] 1788870328.490309 write 0x44d4d9 thru 0x4571cf (0x9cf7 bytes) 000197[1] 1788870328.651668 trunc from 0x964768 to 0x1e6255 (0x77e512 bytes) 000199[1] 1788870328.860253 mapread 0x19d873 thru 0x1a1fbe (0x474c bytes) 000200[0] 1788870328.969902 read 0x0c5072 thru 0x0d08a5 (0xb834 bytes) 000201[0] 1788870329.111746 mapread 0x0c32a3 thru 0x0d0b70 (0xd8ce bytes) 000202[1] 1788870329.177149 punch from 0x18b0bd to 0x193e71 (0x8db5 bytes) 000203[1] 1788870329.276191 mapread 0x030bdf thru 0x0314dc (0x8fe bytes) 000204[1] 1788870329.317764 punch from 0x038b05 to 0x03e1fb (0x56f7 bytes) 000205[0] 1788870329.405469 mapwrite 0x39407b thru 0x397c06 (0x3b8c bytes) 000207[1] 1788870329.799721 read 0x37dfa4 thru 0x38df00 (0xff5d bytes) 000208[0] 1788870329.923195 read 0x2076c3 thru 0x20e60d (0x6f4b bytes) 000209[0] 1788870330.034900 read 0x04e809 thru 0x05258b (0x3d83 bytes) 000211[0] 1788870330.055424 mapwrite 0x0ddb9f thru 0x0e02b0 (0x2712 bytes) 000213[1] 1788870330.294163 mapwrite 0x13583e thru 0x13c6ef (0x6eb2 bytes) 000214[1] 1788870330.641181 mapwrite 0x04dbb4 thru 0x05926f (0xb6bc bytes) 000217[0] 1788870330.955730 trunc from 0x397c07 to 0x2b90b4 (0xdeb52 bytes) 000218[1] 1788870331.069236 read 0x1d6072 thru 0x1e3035 (0xcfc4 bytes) 000219[0] 1788870331.191534 punch from 0x23fb6a to 0x243413 (0x38aa bytes) 000220[1] 1788870331.267448 read 0x0d3ead thru 0x0dcc50 (0x8da4 bytes) 000221[1] 1788870331.330860 punch from 0x118b29 to 0x1254b3 (0xc98b bytes) 000222[1] 1788870331.374680 mapread 0x2a23b9 thru 0x2ad3fa (0xb042 bytes) 000225[1] 1788870331.471197 mapread 0x221570 thru 0x22f55d (0xdfee bytes) 000226[1] 1788870331.548184 write 0x9d27c6 thru 0x9d47f9 (0x2034 bytes) 000227[0] 1788870331.566163 write 0x6dfdf4 thru 0x6e8b89 (0x8d96 bytes) 000229[1] 1788870331.593885 punch from 0x4b0595 to 0x4bfdd7 (0xf843 bytes) 000230[0] 1788870331.617824 mapwrite 0x1835ac thru 0x1836cd (0x122 bytes) 000231[0] 1788870331.727375 mapread 0x44fadc thru 0x4502db (0x800 bytes) 000235[0] 1788870331.893159 mapread 0x0d54e7 thru 0x0dfecb (0xa9e5 bytes) 000237[1] 1788870332.023458 trunc from 0x9d47fa to 0x08f544 (0x9452b5 bytes) 000239[0] 1788870332.174289 trunc from 0x08f545 to 0x3188d4 (0x289390 bytes) 000242[0] 1788870332.276056 trunc from 0x3188d5 to 0x1d3234 (0x1456a0 bytes) 000244[1] 1788870332.312823 mapwrite 0x773870 thru 0x78348f (0xfc20 bytes) 000248[0] 1788870332.702933 mapwrite 0x80a3cd thru 0x812398 (0x7fcc bytes) 000250[0] 1788870332.848878 trunc from 0x812399 to 0x444d05 (0x3cd693 bytes) 000252[0] 1788870332.873136 read 0x40536e thru 0x411fb2 (0xcc45 bytes) 000253[1] 1788870332.922507 trunc from 0x444d06 to 0x8e8e07 (0x4a4102 bytes) 000254[1] 1788870332.993939 write 0x61c516 thru 0x626515 (0xa000 bytes) 000255[1] 1788870333.053192 trunc from 0x8e8e08 to 0x7c3b90 (0x125277 bytes) 000256[1] 1788870333.083113 punch from 0x38ba43 to 0x3922bc (0x687a bytes) 000258[1] 1788870333.090081 punch from 0x45521c to 0x45bdcd (0x6bb2 bytes) 000259[0] 1788870333.097974 read 0x533a8b thru 0x53689a (0x2e10 bytes) 000260[0] 1788870333.155349 trunc from 0x7c3b91 to 0x971aaa (0x1adf1a bytes) 000261[0] 1788870333.238908 mapwrite 0x5b4259 thru 0x5c27fc (0xe5a4 bytes) 000266[1] 1788870333.626901 mapread 0x20ae9e thru 0x2135a6 (0x8709 bytes) 000267[1] 1788870333.731728 trunc from 0x971aab to 0x236cab (0x73adff bytes) 000268[0] 1788870333.822938 punch from 0x1744a1 to 0x17e679 (0xa1d9 bytes) 000270[1] 1788870333.951864 read 0x0a3be3 thru 0x0a47ed (0xc0b bytes) 000271[1] 1788870334.064017 trunc from 0x236cac to 0x8fd600 (0x6c6955 bytes) 000272[1] 1788870334.178092 mapread 0x2fa3fe thru 0x3061ce (0xbdd1 bytes) 000273[1] 1788870334.269580 punch from 0x2b5730 to 0x2b734f (0x1c20 bytes) 000274[0] 1788870334.282340 punch from 0x7fcc6f to 0x80970c (0xca9e bytes) 000275[0] 1788870334.347926 read 0x4b1afa thru 0x4c0ace (0xefd5 bytes) 000276[0] 1788870334.434875 mapread 0x0b5993 thru 0x0c02c3 (0xa931 bytes) 000277[1] 1788870334.498118 punch from 0x6c6561 to 0x6c7f6a (0x1a0a bytes) 000278[1] 1788870334.566855 read 0x46c92c thru 0x472c61 (0x6336 bytes) 000279[1] 1788870334.618293 mapread 0x79bb50 thru 0x79d037 (0x14e8 bytes) 000281[1] 1788870334.712712 read 0x8a85da thru 0x8b2b6f (0xa596 bytes) 000284[1] 1788870334.778701 write 0x76b2c6 thru 0x7712d5 (0x6010 bytes) 000285[0] 1788870334.818462 write 0x45e313 thru 0x463521 (0x520f bytes) 000286[0] 1788870334.891359 write 0x6fc461 thru 0x700f24 (0x4ac4 bytes) 000287[0] 1788870334.936481 read 0x8cd2dc thru 0x8d0a8a (0x37af bytes) 000288[1] 1788870334.956299 mapread 0x51667e thru 0x523aad (0xd430 bytes) 000291[0] 1788870335.018226 write 0x776270 thru 0x77a7c3 (0x4554 bytes) 000292[1] 1788870335.064840 punch from 0x2a3513 to 0x2aeb33 (0xb621 bytes) 000293[1] 1788870335.105113 mapread 0x76232b thru 0x76383b (0x1511 bytes) 000294[0] 1788870335.156053 mapread 0x26bc2b thru 0x26fbf0 (0x3fc6 bytes) 000295[1] 1788870335.190320 write 0x8f9b79 thru 0x906a03 (0xce8b bytes) 000296[0] 1788870335.210812 mapwrite 0x796ed9 thru 0x796ff0 (0x118 bytes) 000299[0] 1788870335.381502 trunc from 0x906a04 to 0x38bdb8 (0x57ac4b bytes) 000301[0] 1788870335.477262 trunc from 0x38bdb9 to 0x5fc531 (0x270779 bytes) 000302[0] 1788870335.503279 mapwrite 0x39debc thru 0x39e131 (0x276 bytes) 000303[1] 1788870335.554657 punch from 0x246880 to 0x24bac8 (0x5249 bytes) 000304[0] 1788870335.599342 read 0x592831 thru 0x59f6b2 (0xce82 bytes) 000305[0] 1788870335.626862 mapwrite 0x62c716 thru 0x637f7a (0xb865 bytes) 000306[0] 1788870335.778877 punch from 0x136bcf to 0x13b001 (0x4433 bytes) 000307[1] 1788870335.815566 trunc from 0x637f7b to 0x8cbdd7 (0x293e5d bytes) 000308[1] 1788870335.908543 read 0x82c7b0 thru 0x837f5a (0xb7ab bytes) 000310[1] 1788870335.959399 write 0x983834 thru 0x98e381 (0xab4e bytes) 000312[0] 1788870335.993240 mapread 0x4ba8d6 thru 0x4c8578 (0xdca3 bytes) 000313[0] 1788870336.029879 punch from 0x299081 to 0x2a10a9 (0x8029 bytes) 000314[0] 1788870336.062990 punch from 0x2d0ac6 to 0x2d6b33 (0x606e bytes) 000315[0] 1788870336.067974 trunc from 0x98e382 to 0x4c497b (0x4c9a06 bytes) 000316[1] 1788870336.131997 punch from 0x1f88d3 to 0x1fdc5b (0x5389 bytes) 000317[0] 1788870336.162051 trunc from 0x4c497c to 0x72e48b (0x269b10 bytes) 000318[1] 1788870336.195724 punch from 0x1ed607 to 0x1f05b8 (0x2fb2 bytes) 000319[1] 1788870336.227562 punch from 0x4ad2b1 to 0x4b205f (0x4daf bytes) 000320[1] 1788870336.277961 mapread 0x54ad84 thru 0x54b9c9 (0xc46 bytes) 000321[0] 1788870336.289847 read 0x313a39 thru 0x319cd6 (0x629e bytes) 000322[0] 1788870336.332486 write 0x27c16b thru 0x286939 (0xa7cf bytes) 000323[1] 1788870336.407266 read 0x1fab11 thru 0x1fb38f (0x87f bytes) 000325[0] 1788870336.465783 punch from 0x57d1b3 to 0x57d6c8 (0x516 bytes) 000328[0] 1788870336.535462 mapread 0x52f960 thru 0x5371da (0x787b bytes) 000331[0] 1788870336.603893 write 0x9d5922 thru 0x9e0e78 (0xb557 bytes) 000333[0] 1788870336.619673 write 0x01d455 thru 0x01f3b0 (0x1f5c bytes) 000334[1] 1788870336.651761 write 0x07a793 thru 0x0880f8 (0xd966 bytes) 000336[1] 1788870336.708129 read 0x4ce421 thru 0x4d9b43 (0xb723 bytes) 000337[1] 1788870336.741096 mapread 0x10d5c8 thru 0x113c88 (0x66c1 bytes) 000338[0] 1788870336.782832 punch from 0x4518f8 to 0x45d283 (0xb98c bytes) 000339[0] 1788870336.812852 write 0x90d17e thru 0x91c77d (0xf600 bytes) 000340[1] 1788870336.864987 trunc from 0x9e0e79 to 0x269c45 (0x777233 bytes) 000341[1] 1788870336.915007 read 0x16db75 thru 0x16fd81 (0x220d bytes) 000342[0] 1788870336.967521 write 0x12af8e thru 0x12c728 (0x179b bytes) 000343[1] 1788870337.006311 mapread 0x1b8ccd thru 0x1b8d95 (0xc9 bytes) 000347[1] 1788870337.040258 read 0x1dd905 thru 0x1ecfab (0xf6a7 bytes) 000349[1] 1788870337.118810 read 0x17507e thru 0x179fc5 (0x4f48 bytes) 000350[0] 1788870337.119389 mapwrite 0x800059 thru 0x801192 (0x113a bytes) 000351[0] 1788870337.276230 read 0x384bd8 thru 0x385557 (0x980 bytes) 000352[0] 1788870337.295102 punch from 0x63d74e to 0x648ec8 (0xb77b bytes) 000353[1] 1788870337.304071 mapread 0x7ef68a thru 0x7fdb25 (0xe49c bytes) 000354[1] 1788870337.519821 mapwrite 0x4e9c96 thru 0x4f8cee (0xf059 bytes) 000355[0] 1788870337.836334 write 0x0a1089 thru 0x0b093e (0xf8b6 bytes) 000356[0] 1788870337.871808 read 0x27f7e6 thru 0x28a64c (0xae67 bytes) 000357[1] 1788870337.917985 punch from 0x3ebd10 to 0x3eee7a (0x316b bytes) 000358[1] 1788870337.996987 read 0x378a40 thru 0x387ace (0xf08f bytes) 000359[1] 1788870338.065525 write 0x6c5b5b thru 0x6cf389 (0x982f bytes) 000360[0] 1788870338.092632 trunc from 0x801193 to 0x928e19 (0x127c87 bytes) 000362[0] 1788870338.163765 read 0x735918 thru 0x741bc0 (0xc2a9 bytes) 000363[1] 1788870338.216284 write 0x38fdaa thru 0x3947d3 (0x4a2a bytes) 000364[1] 1788870338.276923 write 0x354093 thru 0x355369 (0x12d7 bytes) 000367[1] 1788870338.347144 mapwrite 0x1440ad thru 0x1457fe (0x1752 bytes) 000368[1] 1788870338.445272 mapwrite 0x3bf59a thru 0x3ca84d (0xb2b4 bytes) 000369[1] 1788870338.626087 trunc from 0x928e1a to 0x30b0f1 (0x61dd28 bytes) 000370[1] 1788870338.662996 mapread 0x0fc512 thru 0x10b332 (0xee21 bytes) 000373[1] 1788870338.778152 write 0x1d0553 thru 0x1d1599 (0x1047 bytes) 000374[0] 1788870338.796465 punch from 0x179b26 to 0x18184f (0x7d2a bytes) 000375[0] 1788870338.845906 punch from 0x1d1ced to 0x1d6bc6 (0x4eda bytes) 000379[0] 1788870338.949160 trunc from 0x30b0f2 to 0x4081c2 (0xfd0d1 bytes) 000382[1] 1788870339.060461 mapwrite 0x2bae4c thru 0x2c3bbb (0x8d70 bytes) 000383[1] 1788870339.198936 write 0x00c81d thru 0x01b91c (0xf100 bytes) 000385[0] 1788870339.251021 trunc from 0x4081c3 to 0x314c85 (0xf353d bytes) 000387[1] 1788870339.308062 mapread 0x16ed3a thru 0x1715c1 (0x2888 bytes) 000388[1] 1788870339.348489 trunc from 0x314c86 to 0x94cc63 (0x637fde bytes) 000390[1] 1788870339.438520 read 0x6bd5a8 thru 0x6befdb (0x1a34 bytes) 000392[0] 1788870339.452255 mapwrite 0x0f557f thru 0x105463 (0xfee5 bytes) 000394[1] 1788870339.577225 write 0x30adbc thru 0x30db00 (0x2d45 bytes) 000395[1] 1788870339.599277 mapread 0x35ac1f thru 0x3649fe (0x9de0 bytes) 000396[0] 1788870339.632302 mapwrite 0x37241e thru 0x37cc01 (0xa7e4 bytes) 000398[0] 1788870339.783396 mapread 0x76c0fb thru 0x77b4a8 (0xf3ae bytes) 000399[1] 1788870339.843074 trunc from 0x94cc64 to 0x66d551 (0x2df712 bytes) 000400[1] 1788870339.889685 read 0x326df6 thru 0x32f553 (0x875e bytes) 000403[1] 1788870339.921336 mapread 0x02313c thru 0x02a74b (0x7610 bytes) 000405[1] 1788870339.986054 mapread 0x0c6e31 thru 0x0d3f5a (0xd12a bytes) 000407[1] 1788870340.014735 trunc from 0x66d552 to 0x5c487f (0xa8cd2 bytes) 000408[0] 1788870340.042432 mapread 0x3665fa thru 0x366ecc (0x8d3 bytes) 000410[0] 1788870340.148683 mapwrite 0x8cff4e thru 0x8dbef5 (0xbfa8 bytes) 000411[0] 1788870340.476445 punch from 0x1b650e to 0x1c0a6f (0xa562 bytes) 000412[0] 1788870340.492878 punch from 0x89aee4 to 0x8a01b4 (0x52d1 bytes) 000414[0] 1788870340.502987 trunc from 0x8dbef6 to 0x2fc50a (0x5df9eb bytes) 000415[1] 1788870340.595183 trunc from 0x2fc50b to 0x94812c (0x64bc22 bytes) 000416[0] 1788870340.680907 write 0x9cd764 thru 0x9d3cc0 (0x655d bytes) 000418[1] 1788870340.772912 mapread 0x625d33 thru 0x63006c (0xa33a bytes) 000420[1] 1788870340.883954 write 0x6add57 thru 0x6bc5a7 (0xe851 bytes) 000421[1] 1788870340.918729 mapread 0x0b8f3c thru 0x0bfbc4 (0x6c89 bytes) truncating to largest ever: 0x9ff066 000423[1] 1788870340.992895 trunc from 0x9d3cc1 to 0x9ff065 (0x2b3a5 bytes) 000424[1] 1788870341.042947 punch from 0x6142e9 to 0x616a32 (0x274a bytes) 000425[1] 1788870341.074823 mapwrite 0x2c7edb thru 0x2d3855 (0xb97b bytes) 000426[1] 1788870341.291420 punch from 0x6b512d to 0x6c3bf6 (0xeaca bytes) 000427[1] 1788870341.343555 trunc from 0x9ff066 to 0x19d369 (0x861cfc bytes) 000428[1] 1788870341.411231 write 0x7747bc thru 0x781810 (0xd055 bytes) 000432[0] 1788870341.472509 trunc from 0x781811 to 0x19bb48 (0x5e5cc8 bytes) 000433[1] 1788870341.616652 trunc from 0x19bb49 to 0x96a4d9 (0x7ce991 bytes) 000436[1] 1788870341.728231 mapread 0x89b91f thru 0x8a1b07 (0x61e9 bytes) 000437[0] 1788870341.777957 trunc from 0x96a4da to 0x4ffd9e (0x46a73b bytes) 000441[1] 1788870341.957792 trunc from 0x4ffd9f to 0x5f2624 (0xf2886 bytes) 000445[0] 1788870342.094480 mapread 0x5d28ce thru 0x5d8c33 (0x6366 bytes) 000448[0] 1788870342.165567 write 0x0d4672 thru 0x0dbf56 (0x78e5 bytes) 000452[0] 1788870342.302904 mapread 0x59625f thru 0x5a24b0 (0xc252 bytes) 000454[0] 1788870342.326460 trunc from 0x5f2625 to 0x8a6021 (0x2b39fd bytes) 000455[1] 1788870342.409460 write 0x3e7f00 thru 0x3ee4fb (0x65fc bytes) 000456[0] 1788870342.464266 mapread 0x490417 thru 0x49c547 (0xc131 bytes) 000459[0] 1788870342.606863 trunc from 0x8a6022 to 0x2e4c3c (0x5c13e5 bytes) 000460[0] 1788870342.694412 read 0x0e2a82 thru 0x0e6eab (0x442a bytes) 000461[1] 1788870342.768961 punch from 0x18448c to 0x192f9a (0xeb0f bytes) 000462[0] 1788870342.830961 mapwrite 0x00178f thru 0x00aecb (0x973d bytes) 000464[1] 1788870343.104092 read 0x13d69e thru 0x1416d2 (0x4035 bytes) 000466[0] 1788870343.281515 write 0x474826 thru 0x474a6c (0x247 bytes) 000467[1] 1788870343.289637 read 0x299a94 thru 0x2a0054 (0x65c1 bytes) 000468[0] 1788870343.312039 mapread 0x0860c7 thru 0x091d20 (0xbc5a bytes) 000469[0] 1788870343.403959 read 0x08203d thru 0x08b643 (0x9607 bytes) 000470[1] 1788870343.419760 mapwrite 0x4d36e3 thru 0x4dc254 (0x8b72 bytes) 000471[0] 1788870343.774682 punch from 0x3c1c83 to 0x3c5423 (0x37a1 bytes) 000472[0] 1788870343.912203 mapread 0x10e508 thru 0x118ad5 (0xa5ce bytes) 000473[1] 1788870344.022348 trunc from 0x4dc255 to 0x3aa528 (0x131d2c bytes) 000476[1] 1788870344.176418 trunc from 0x3aa529 to 0x870125 (0x4c5bfd bytes) 000478[1] 1788870344.211736 trunc from 0x870126 to 0x9019b7 (0x91892 bytes) 000479[0] 1788870344.273863 read 0x0af4ee thru 0x0af671 (0x184 bytes) 000481[0] 1788870344.339857 mapwrite 0x17d21c thru 0x183d14 (0x6af9 bytes) 000482[0] 1788870344.480177 trunc from 0x9019b8 to 0x6c8470 (0x239547 bytes) 000485[0] 1788870344.562365 punch from 0x69c992 to 0x69ed39 (0x23a8 bytes) 000486[0] 1788870344.595060 mapwrite 0x15f9ec thru 0x16a15f (0xa774 bytes) 000489[1] 1788870344.713456 mapread 0x3a42be thru 0x3a92e6 (0x5029 bytes) 000490[0] 1788870344.778898 write 0x5cd9a6 thru 0x5d77ca (0x9e25 bytes) 000491[1] 1788870344.796350 mapwrite 0x3e5557 thru 0x3ea568 (0x5012 bytes) 000492[1] 1788870344.945999 punch from 0x4e5587 to 0x4ef9cc (0xa446 bytes) 000494[0] 1788870345.134983 write 0x681d13 thru 0x68f177 (0xd465 bytes) 000498[1] 1788870345.269570 mapwrite 0x941abf thru 0x941e24 (0x366 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 345 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[0] 1788870346.898366 write_OD 0x82d000 thru 0x839fff (0xd000 bytes) 000007[0] 1788870346.962223 punch from 0x050343 to 0x05dcdd (0xd99b bytes) 000009[1] 1788870347.001738 mapwrite 0x6fe000 thru 0x709c42 (0xbc43 bytes) 000011[0] 1788870347.125918 mapread 0x4a1000 thru 0x4a9242 (0x8243 bytes) 000012[0] 1788870347.180009 read_OD 0x2bf000 thru 0x2cdfff (0xf000 bytes) 000016[0] 1788870347.195750 mapwrite 0x2f7000 thru 0x301338 (0xa339 bytes) 000019[0] 1788870347.378797 punch from 0x7f9a2a to 0x808e79 (0xf450 bytes) 000020[0] 1788870347.387234 mapread 0x5ba000 thru 0x5bb939 (0x193a bytes) 000026[0] 1788870347.462491 mapread 0x10e000 thru 0x119f89 (0xbf8a bytes) 000028[0] 1788870347.547790 punch from 0x6f05ec to 0x6fffb5 (0xf9ca bytes) 000029[1] 1788870347.582530 punch from 0x56314a to 0x568162 (0x5019 bytes) 000031[1] 1788870347.689843 mapread 0x726000 thru 0x72b8d6 (0x58d7 bytes) 000032[1] 1788870347.705923 punch from 0x63f87e to 0x64eb6c (0xf2ef bytes) truncating to largest ever: 0x577b63 000034[1] 1788870347.746681 trunc from 0x83a000 to 0x577b62 (0x2c249d bytes) 000037[1] 1788870347.820717 trunc from 0x577b63 to 0x5027e8 (0x7537a bytes) 000038[1] 1788870347.866555 trunc from 0x5027e9 to 0x25ed38 (0x2a3ab0 bytes) 000040[1] 1788870347.966450 read 0x131000 thru 0x13e433 (0xd434 bytes) 000041[1] 1788870348.034186 trunc from 0x25ed39 to 0x35ea57 (0xffd1f bytes) 000042[0] 1788870348.108674 read_OD 0x0c0000 thru 0x0c9fff (0xa000 bytes) 000043[1] 1788870348.177419 write 0x5f5000 thru 0x5f5284 (0x285 bytes) 000044[1] 1788870348.220554 read 0x39b000 thru 0x3a0242 (0x5243 bytes) 000046[0] 1788870348.282232 mapread 0x0cc000 thru 0x0d57a6 (0x97a7 bytes) 000047[0] 1788870348.445737 mapread 0x2e2000 thru 0x2e6c5c (0x4c5d bytes) 000048[1] 1788870348.446379 trunc from 0x5f5285 to 0x276636 (0x37ec4e bytes) 000051[0] 1788870348.531013 write_OD 0x749000 thru 0x74bfff (0x3000 bytes) 000052[1] 1788870348.586058 punch from 0x1552a4 to 0x159d4f (0x4aac bytes) 000053[1] 1788870348.606724 mapwrite 0x804000 thru 0x807b50 (0x3b51 bytes) 000055[0] 1788870348.756825 mapwrite 0x9a6000 thru 0x9ac850 (0x6851 bytes) 000056[0] 1788870348.984735 read_OD 0x940000 thru 0x94cfff (0xd000 bytes) 000059[1] 1788870349.017529 write 0x2d2000 thru 0x2d43b4 (0x23b5 bytes) truncating to largest ever: 0x9af3df 000060[0] 1788870349.034842 trunc from 0x9ac851 to 0x9af3de (0x2b8e bytes) 000063[0] 1788870349.069044 mapread 0x0da000 thru 0x0dab03 (0xb04 bytes) 000064[1] 1788870349.092225 mapwrite 0x93d000 thru 0x942653 (0x5654 bytes) 000065[0] 1788870349.243420 read_OD 0x894000 thru 0x8a0fff (0xd000 bytes) 000068[1] 1788870349.293966 read 0x227000 thru 0x22fbea (0x8beb bytes) 000071[1] 1788870349.313926 mapread 0x0b0000 thru 0x0bb093 (0xb094 bytes) 000073[1] 1788870349.381670 punch from 0x204a44 to 0x20c312 (0x78cf bytes) 000074[0] 1788870349.412768 mapwrite 0x91c000 thru 0x9244f5 (0x84f6 bytes) 000075[0] 1788870349.563171 punch from 0x952b84 to 0x9588ab (0x5d28 bytes) 000076[1] 1788870349.591554 mapwrite 0x4a1000 thru 0x4a556a (0x456b bytes) 000078[1] 1788870349.757926 write 0x2bf000 thru 0x2ccf2a (0xdf2b bytes) 000079[1] 1788870349.770210 punch from 0x58e5eb to 0x58ebfa (0x610 bytes) 000080[1] 1788870349.779036 mapread 0x26d000 thru 0x2750d3 (0x80d4 bytes) 000081[1] 1788870349.837266 trunc from 0x9af3df to 0x58752e (0x427eb0 bytes) 000082[1] 1788870349.908982 trunc from 0x58752f to 0x1d7305 (0x3b0229 bytes) 000083[0] 1788870349.951109 trunc from 0x1d7306 to 0x418bd6 (0x2418d1 bytes) 000087[0] 1788870350.040787 trunc from 0x418bd7 to 0x8ac321 (0x49374b bytes) 000088[1] 1788870350.107113 punch from 0x7699f6 to 0x776038 (0xc643 bytes) 000089[1] 1788870350.166394 mapwrite 0x4d4000 thru 0x4d6e52 (0x2e53 bytes) 000092[0] 1788870350.267192 mapread 0x382000 thru 0x3904f2 (0xe4f3 bytes) 000093[1] 1788870350.315871 write 0x0c2000 thru 0x0d172f (0xf730 bytes) 000094[0] 1788870350.369515 read_OD 0x13c000 thru 0x145fff (0xa000 bytes) 000095[1] 1788870350.411435 punch from 0x5c57e7 to 0x5ced36 (0x9550 bytes) 000098[0] 1788870350.447719 punch from 0x87eb89 to 0x8842d5 (0x574d bytes) 000099[1] 1788870350.489098 punch from 0x82828e to 0x835ee3 (0xdc56 bytes) 000100[0] 1788870350.503413 write_OD 0x989000 thru 0x989fff (0x1000 bytes) 000101[0] 1788870350.554554 mapwrite 0x48d000 thru 0x49b314 (0xe315 bytes) 000103[1] 1788870350.682198 mapwrite 0x192000 thru 0x1983d8 (0x63d9 bytes) 000105[1] 1788870350.797140 mapread 0x5b8000 thru 0x5c6602 (0xe603 bytes) 000106[1] 1788870350.830066 mapwrite 0x125000 thru 0x12fb5f (0xab60 bytes) 000107[0] 1788870350.957328 mapread 0x066000 thru 0x068b07 (0x2b08 bytes) 000108[0] 1788870351.041670 write_OD 0x872000 thru 0x873fff (0x2000 bytes) 000109[0] 1788870351.203982 read_OD 0x3c7000 thru 0x3d4fff (0xe000 bytes) 000110[0] 1788870351.240722 write_OD 0x456000 thru 0x456fff (0x1000 bytes) 000111[0] 1788870351.307005 punch from 0x0d956f to 0x0dc483 (0x2f15 bytes) 000112[0] 1788870351.336164 read_OD 0x7db000 thru 0x7ddfff (0x3000 bytes) 000113[1] 1788870351.353322 mapwrite 0x94e000 thru 0x9546bf (0x66c0 bytes) 000115[0] 1788870351.435869 mapread 0x5dd000 thru 0x5e91f4 (0xc1f5 bytes) truncating to largest ever: 0x9e32c7 000116[0] 1788870351.482779 trunc from 0x98a000 to 0x9e32c6 (0x592c7 bytes) 000117[1] 1788870351.516716 mapread 0x509000 thru 0x50db36 (0x4b37 bytes) 000120[1] 1788870351.562313 read 0x999000 thru 0x99ad0b (0x1d0c bytes) 000121[1] 1788870351.567110 read 0x97e000 thru 0x988128 (0xa129 bytes) 000122[1] 1788870351.574297 write 0x909000 thru 0x9158be (0xc8bf bytes) 000123[0] 1788870351.607882 write_OD 0x357000 thru 0x35efff (0x8000 bytes) 000124[0] 1788870351.694103 trunc from 0x9e32c7 to 0x1ed93d (0x7f5989 bytes) 000125[0] 1788870351.761511 trunc from 0x1ed93e to 0x3f1cb0 (0x204373 bytes) 000126[1] 1788870351.837553 mapwrite 0x908000 thru 0x909bd4 (0x1bd5 bytes) 000127[0] 1788870351.946585 punch from 0x3a05ed to 0x3abf1b (0xb92f bytes) 000130[0] 1788870351.969320 trunc from 0x909bd5 to 0x240cd8 (0x6c8efc bytes) 000131[0] 1788870352.018340 write_OD 0x0ed000 thru 0x0fafff (0xe000 bytes) 000134[0] 1788870352.099880 mapread 0x0c6000 thru 0x0c7608 (0x1609 bytes) 000135[0] 1788870352.112751 trunc from 0x240cd9 to 0x7a6209 (0x565531 bytes) 000136[0] 1788870352.171174 read_OD 0x79d000 thru 0x7a0fff (0x4000 bytes) 000139[1] 1788870352.209726 punch from 0x3307df to 0x331991 (0x11b3 bytes) 000142[1] 1788870352.310306 mapread 0x0f6000 thru 0x10447b (0xe47c bytes) 000143[0] 1788870352.453119 mapwrite 0x421000 thru 0x42e8ed (0xd8ee bytes) 000145[1] 1788870352.698852 mapwrite 0x3c6000 thru 0x3c6181 (0x182 bytes) 000146[0] 1788870352.912865 write_OD 0x910000 thru 0x91cfff (0xd000 bytes) 000147[0] 1788870352.970429 mapwrite 0x755000 thru 0x764ff3 (0xfff4 bytes) 000148[0] 1788870353.081571 mapwrite 0x8b1000 thru 0x8b1729 (0x72a bytes) 000149[0] 1788870353.135363 mapread 0x5bc000 thru 0x5c08fb (0x48fc bytes) 000152[0] 1788870353.183977 mapread 0x04c000 thru 0x05792d (0xb92e bytes) 000154[0] 1788870353.253919 trunc from 0x91d000 to 0x572538 (0x3aaac7 bytes) 000155[0] 1788870353.290509 write_OD 0x424000 thru 0x428fff (0x5000 bytes) 000156[0] 1788870353.337586 write_OD 0x5e6000 thru 0x5e7fff (0x2000 bytes) 000157[0] 1788870353.395831 trunc from 0x5e8000 to 0x8ea7c8 (0x3027c9 bytes) 000159[0] 1788870353.442911 mapwrite 0x1d0000 thru 0x1de842 (0xe843 bytes) 000161[1] 1788870353.550615 write_OD 0x83d000 thru 0x844fff (0x8000 bytes) 000162[1] 1788870353.588111 mapwrite 0x484000 thru 0x487074 (0x3075 bytes) 000163[0] 1788870353.636970 punch from 0x82860f to 0x830488 (0x7e7a bytes) 000164[0] 1788870353.650647 mapread 0x4ce000 thru 0x4d60f9 (0x80fa bytes) 000165[1] 1788870353.667310 mapread 0x743000 thru 0x74d7a6 (0xa7a7 bytes) 000166[1] 1788870353.710522 trunc from 0x8ea7c9 to 0x1fbc3f (0x6eeb89 bytes) 000167[0] 1788870353.757211 trunc from 0x1fbc40 to 0x432cde (0x23709f bytes) 000169[1] 1788870353.827246 read_OD 0x171000 thru 0x177fff (0x7000 bytes) 000170[0] 1788870353.845342 trunc from 0x432cdf to 0x7d7bf2 (0x3a4f14 bytes) 000171[1] 1788870353.863295 punch from 0x2e4626 to 0x2ef982 (0xb35d bytes) 000173[1] 1788870353.874099 punch from 0x530c87 to 0x5319da (0xd54 bytes) 000175[0] 1788870353.913356 write_OD 0x95b000 thru 0x967fff (0xd000 bytes) 000176[1] 1788870353.959633 mapwrite 0x154000 thru 0x1601cb (0xc1cc bytes) 000177[1] 1788870354.040214 mapwrite 0x8ef000 thru 0x8fe998 (0xf999 bytes) 000178[0] 1788870354.110650 read_OD 0x7b0000 thru 0x7bdfff (0xe000 bytes) 000180[0] 1788870354.129903 write_OD 0x46b000 thru 0x46ffff (0x5000 bytes) 000182[0] 1788870354.176106 punch from 0x503d2c to 0x51240f (0xe6e4 bytes) 000183[0] 1788870354.194896 mapwrite 0x8c3000 thru 0x8ca329 (0x732a bytes) 000185[0] 1788870354.278379 mapwrite 0x790000 thru 0x791789 (0x178a bytes) 000186[0] 1788870354.341269 punch from 0x2920df to 0x29abfc (0x8b1e bytes) 000187[0] 1788870354.345507 mapwrite 0x785000 thru 0x79323f (0xe240 bytes) 000188[0] 1788870354.421284 mapread 0x5f5000 thru 0x600ccc (0xbccd bytes) 000190[0] 1788870354.447785 mapread 0x7d3000 thru 0x7e007d (0xd07e bytes) 000191[1] 1788870354.487142 write_OD 0x95d000 thru 0x966fff (0xa000 bytes) 000192[1] 1788870354.532047 read_OD 0x6c3000 thru 0x6d1fff (0xf000 bytes) 000193[1] 1788870354.553459 mapread 0x0ef000 thru 0x0f9e28 (0xae29 bytes) 000194[1] 1788870354.574961 mapwrite 0x956000 thru 0x962e7e (0xce7f bytes) 000196[1] 1788870354.660361 punch from 0x4b1b99 to 0x4b21e5 (0x64d bytes) 000198[1] 1788870354.677271 mapread 0x354000 thru 0x363edb (0xfedc bytes) 000202[0] 1788870354.758083 mapread 0x3bf000 thru 0x3c743c (0x843d bytes) 000209[1] 1788870354.825408 write_OD 0x6c5000 thru 0x6cefff (0xa000 bytes) 000211[1] 1788870354.882510 read_OD 0x88c000 thru 0x896fff (0xb000 bytes) 000213[1] 1788870354.889998 write_OD 0x423000 thru 0x42cfff (0xa000 bytes) 000215[1] 1788870354.937067 trunc from 0x968000 to 0x973b4d (0xbb4e bytes) 000217[1] 1788870354.973785 read_OD 0x0df000 thru 0x0dffff (0x1000 bytes) 000218[1] 1788870354.989321 mapwrite 0x734000 thru 0x735baf (0x1bb0 bytes) 000219[0] 1788870355.050147 write_OD 0x2c6000 thru 0x2c7fff (0x2000 bytes) 000220[1] 1788870355.106522 write_OD 0x08f000 thru 0x09cfff (0xe000 bytes) 000223[1] 1788870355.172079 trunc from 0x973b4e to 0x348e4c (0x62ad01 bytes) 000224[1] 1788870355.209980 punch from 0x2ef434 to 0x2f7e93 (0x8a60 bytes) 000225[1] 1788870355.228903 punch from 0x1aac90 to 0x1b1e9f (0x7210 bytes) 000226[0] 1788870355.233568 mapread 0x1f6000 thru 0x20321d (0xd21e bytes) 000227[1] 1788870355.294578 mapread 0x088000 thru 0x09678e (0xe78f bytes) 000229[0] 1788870355.330472 read_OD 0x056000 thru 0x05dfff (0x8000 bytes) 000231[0] 1788870355.346076 mapread 0x314000 thru 0x3157e0 (0x17e1 bytes) 000232[0] 1788870355.399827 mapread 0x260000 thru 0x26f403 (0xf404 bytes) 000233[0] 1788870355.400509 mapread 0x25d000 thru 0x26b2cc (0xe2cd bytes) 000237[1] 1788870355.401907 read_OD 0x2f1000 thru 0x2fbfff (0xb000 bytes) 000238[0] 1788870355.414122 punch from 0x328186 to 0x32eb70 (0x69eb bytes) 000239[0] 1788870355.436279 trunc from 0x348e4d to 0x92a48e (0x5e1642 bytes) 000243[0] 1788870355.524170 write_OD 0x912000 thru 0x919fff (0x8000 bytes) 000244[1] 1788870355.577319 trunc from 0x92a48f to 0x932c40 (0x87b2 bytes) 000246[0] 1788870355.621304 mapwrite 0x027000 thru 0x027bf4 (0xbf5 bytes) 000247[1] 1788870355.706256 read_OD 0x367000 thru 0x368fff (0x2000 bytes) 000248[0] 1788870355.729368 read_OD 0x868000 thru 0x870fff (0x9000 bytes) 000250[0] 1788870355.741288 mapwrite 0x06d000 thru 0x074ec1 (0x7ec2 bytes) 000251[1] 1788870355.829164 read_OD 0x463000 thru 0x467fff (0x5000 bytes) 000252[0] 1788870355.860062 read_OD 0x246000 thru 0x249fff (0x4000 bytes) 000253[0] 1788870355.874818 mapwrite 0x17c000 thru 0x1838fe (0x78ff bytes) 000255[0] 1788870355.979394 punch from 0x1fae12 to 0x209fab (0xf19a bytes) 000257[0] 1788870356.004468 mapread 0x34d000 thru 0x357c5c (0xac5d bytes) 000258[0] 1788870356.036252 read_OD 0x82d000 thru 0x830fff (0x4000 bytes) 000261[1] 1788870356.066636 read_OD 0x86a000 thru 0x876fff (0xd000 bytes) 000263[1] 1788870356.080499 trunc from 0x932c41 to 0x436aaf (0x4fc191 bytes) 000264[0] 1788870356.134730 punch from 0x0f4b92 to 0x0f53eb (0x85a bytes) 000267[0] 1788870356.186539 mapwrite 0x007000 thru 0x008b28 (0x1b29 bytes) 000268[0] 1788870356.257390 read_OD 0x11e000 thru 0x124fff (0x7000 bytes) 000269[1] 1788870356.281094 mapwrite 0x033000 thru 0x03dd5d (0xad5e bytes) truncating to largest ever: 0x9eba55 000270[0] 1788870356.380652 trunc from 0x436ab0 to 0x9eba54 (0x5b4fa5 bytes) 000272[1] 1788870356.411463 trunc from 0x9eba55 to 0x3602bb (0x68b799 bytes) 000274[1] 1788870356.466174 punch from 0x279eed to 0x27e691 (0x47a5 bytes) 000275[0] 1788870356.472961 punch from 0x178c67 to 0x1810d0 (0x846a bytes) 000277[1] 1788870356.489791 punch from 0x024637 to 0x024b55 (0x51f bytes) 000278[0] 1788870356.508417 mapread 0x2fc000 thru 0x30252e (0x652f bytes) 000279[1] 1788870356.551037 punch from 0x00cc1d to 0x0116ee (0x4ad2 bytes) 000280[0] 1788870356.568258 mapread 0x0c9000 thru 0x0d1fe6 (0x8fe7 bytes) 000281[1] 1788870356.606132 mapwrite 0x608000 thru 0x60ea0d (0x6a0e bytes) 000283[0] 1788870356.772402 punch from 0x2e1d39 to 0x2ecd0d (0xafd5 bytes) 000285[1] 1788870356.825233 write_OD 0x2a5000 thru 0x2a8fff (0x4000 bytes) 000286[0] 1788870356.882023 trunc from 0x60ea0e to 0x798e5a (0x18a44d bytes) 000287[1] 1788870356.927732 read_OD 0x01a000 thru 0x01afff (0x1000 bytes) 000290[0] 1788870356.958774 mapwrite 0x441000 thru 0x444cde (0x3cdf bytes) 000292[0] 1788870357.064343 read_OD 0x746000 thru 0x747fff (0x2000 bytes) 000297[1] 1788870357.094830 read_OD 0x433000 thru 0x43cfff (0xa000 bytes) 000298[0] 1788870357.107333 read_OD 0x0f4000 thru 0x102fff (0xf000 bytes) 000303[0] 1788870357.130087 read_OD 0x128000 thru 0x135fff (0xe000 bytes) 000304[0] 1788870357.139697 mapwrite 0x98b000 thru 0x99247c (0x747d bytes) 000308[0] 1788870357.336411 read_OD 0x064000 thru 0x06dfff (0xa000 bytes) 000313[0] 1788870357.363358 trunc from 0x99247d to 0x2ecaa7 (0x6a59d5 bytes) 000315[1] 1788870357.421064 read_OD 0x0cf000 thru 0x0dafff (0xc000 bytes) 000316[1] 1788870357.452708 read_OD 0x2c5000 thru 0x2cbfff (0x7000 bytes) 000318[1] 1788870357.465791 mapread 0x1f3000 thru 0x20096a (0xd96b bytes) 000319[0] 1788870357.537085 mapwrite 0x6ac000 thru 0x6b0933 (0x4934 bytes) 000320[0] 1788870357.732044 trunc from 0x6b0934 to 0x7e9ecf (0x13959c bytes) 000321[0] 1788870357.765231 read_OD 0x3fd000 thru 0x408fff (0xc000 bytes) 000322[0] 1788870357.783061 mapwrite 0x2d0000 thru 0x2dea6d (0xea6e bytes) 000324[0] 1788870357.964880 read_OD 0x7da000 thru 0x7e1fff (0x8000 bytes) 000325[0] 1788870357.995291 punch from 0x138ad5 to 0x13dd0a (0x5236 bytes) 000327[0] 1788870358.060472 punch from 0x44cccf to 0x45a7ac (0xdade bytes) 000329[1] 1788870358.114183 mapread 0x233000 thru 0x233bcc (0xbcd bytes) 000330[0] 1788870358.179280 write_OD 0x820000 thru 0x823fff (0x4000 bytes) 000333[0] 1788870358.249269 write_OD 0x853000 thru 0x85bfff (0x9000 bytes) 000334[1] 1788870358.300441 mapread 0x1a1000 thru 0x1acf20 (0xbf21 bytes) 000335[1] 1788870358.380809 write_OD 0x438000 thru 0x439fff (0x2000 bytes) 000336[0] 1788870358.468989 mapread 0x740000 thru 0x747370 (0x7371 bytes) 000337[0] 1788870358.493706 write_OD 0x128000 thru 0x136fff (0xf000 bytes) 000339[0] 1788870358.594469 trunc from 0x85c000 to 0x7932ec (0xc8d13 bytes) 000341[0] 1788870358.643793 trunc from 0x7932ed to 0x92dbe1 (0x19a8f5 bytes) 000342[0] 1788870358.667571 punch from 0x8251ea to 0x82c9c7 (0x77de bytes) 000343[0] 1788870358.692312 punch from 0x71278f to 0x717d26 (0x5598 bytes) 000345[0] 1788870358.712953 mapread 0x8b7000 thru 0x8c5930 (0xe931 bytes) 000347[0] 1788870358.749735 trunc from 0x92dbe2 to 0x982b1f (0x54f3e bytes) 000348[1] 1788870358.775804 mapwrite 0x30d000 thru 0x319a27 (0xca28 bytes) 000350[0] 1788870359.129832 write_OD 0x8c4000 thru 0x8c5fff (0x2000 bytes) 000354[0] 1788870359.213098 read_OD 0x956000 thru 0x95cfff (0x7000 bytes) 000356[0] 1788870359.224946 write_OD 0x675000 thru 0x67ffff (0xb000 bytes) 000357[1] 1788870359.308551 trunc from 0x982b20 to 0x60e764 (0x3743bb bytes) 000359[1] 1788870359.366194 read_OD 0x2f9000 thru 0x306fff (0xe000 bytes) 000362[0] 1788870359.392259 mapwrite 0x716000 thru 0x71b115 (0x5116 bytes) 000363[1] 1788870359.502213 trunc from 0x71b116 to 0x2468e5 (0x4d4830 bytes) 000364[0] 1788870359.548004 read_OD 0x198000 thru 0x1a1fff (0xa000 bytes) 000366[1] 1788870359.577383 read_OD 0x1f0000 thru 0x1f6fff (0x7000 bytes) 000368[1] 1788870359.589036 write_OD 0x004000 thru 0x00bfff (0x8000 bytes) 000370[0] 1788870359.715008 mapread 0x22c000 thru 0x22ea22 (0x2a23 bytes) 000371[0] 1788870359.819392 mapread 0x120000 thru 0x12c296 (0xc297 bytes) 000372[0] 1788870359.820051 trunc from 0x2468e6 to 0x323b8a (0xdd2a5 bytes) 000374[1] 1788870359.960335 mapread 0x217000 thru 0x219f6f (0x2f70 bytes) 000377[0] 1788870360.038576 write_OD 0x2fe000 thru 0x301fff (0x4000 bytes) 000378[1] 1788870360.092905 mapread 0x164000 thru 0x164675 (0x676 bytes) 000380[1] 1788870360.105871 mapread 0x217000 thru 0x222c1a (0xbc1b bytes) 000383[1] 1788870360.162826 write_OD 0x619000 thru 0x624fff (0xc000 bytes) 000384[0] 1788870360.225063 punch from 0x0f1b9e to 0x0f5e06 (0x4269 bytes) 000385[1] 1788870360.263457 trunc from 0x625000 to 0x43cc15 (0x1e83ea bytes) 000386[0] 1788870360.374467 trunc from 0x43cc16 to 0x9e6e09 (0x5aa1f4 bytes) 000388[1] 1788870360.516074 trunc from 0x9e6e0a to 0x6d9128 (0x30dce1 bytes) 000395[0] 1788870360.747193 trunc from 0x6d9129 to 0x2680c2 (0x471066 bytes) 000398[0] 1788870360.934825 write_OD 0x84e000 thru 0x85afff (0xd000 bytes) 000399[1] 1788870361.103721 punch from 0x01fcab to 0x021d57 (0x20ad bytes) 000400[0] 1788870361.190337 punch from 0x14b418 to 0x152052 (0x6c3b bytes) 000402[1] 1788870361.259209 punch from 0x3dc669 to 0x3ddd9d (0x1735 bytes) 000405[0] 1788870361.350782 read_OD 0x208000 thru 0x20bfff (0x4000 bytes) 000406[0] 1788870361.390756 mapwrite 0x554000 thru 0x56014f (0xc150 bytes) 000411[1] 1788870361.613833 mapread 0x693000 thru 0x69e0c5 (0xb0c6 bytes) 000412[1] 1788870361.703890 mapwrite 0x05c000 thru 0x0681ae (0xc1af bytes) 000414[1] 1788870362.043837 read_OD 0x710000 thru 0x718fff (0x9000 bytes) 000415[0] 1788870362.072327 trunc from 0x85b000 to 0x34860e (0x5129f1 bytes) 000418[0] 1788870362.152497 mapread 0x153000 thru 0x159044 (0x6045 bytes) 000419[1] 1788870362.193890 punch from 0x03af22 to 0x04ad47 (0xfe26 bytes) 000421[1] 1788870362.250028 mapread 0x321000 thru 0x32d6ad (0xc6ae bytes) 000423[0] 1788870362.311011 mapread 0x2a4000 thru 0x2a902e (0x502f bytes) 000424[0] 1788870362.374165 read_OD 0x206000 thru 0x20bfff (0x6000 bytes) 000425[1] 1788870362.384883 read_OD 0x284000 thru 0x286fff (0x3000 bytes) 000426[0] 1788870362.393837 trunc from 0x34860f to 0x3502ec (0x7cde bytes) 000427[0] 1788870362.449347 mapread 0x25d000 thru 0x268d90 (0xbd91 bytes) 000428[0] 1788870362.508338 read_OD 0x07c000 thru 0x080fff (0x5000 bytes) 000429[1] 1788870362.542343 trunc from 0x3502ed to 0x55ac2e (0x20a942 bytes) 000430[0] 1788870362.584279 mapread 0x41e000 thru 0x42dcf5 (0xfcf6 bytes) 000431[0] 1788870362.730469 mapread 0x53f000 thru 0x544969 (0x596a bytes) 000432[1] 1788870362.732255 mapwrite 0x26b000 thru 0x275f05 (0xaf06 bytes) 000433[0] 1788870362.926908 mapwrite 0x087000 thru 0x0902a4 (0x92a5 bytes) 000434[0] 1788870363.152007 trunc from 0x55ac2f to 0x831266 (0x2d6638 bytes) 000438[1] 1788870363.299891 trunc from 0x831267 to 0x9e4625 (0x1b33bf bytes) 000439[1] 1788870363.372997 punch from 0x8963a4 to 0x89d556 (0x71b3 bytes) 000440[0] 1788870363.391098 read_OD 0x1a1000 thru 0x1adfff (0xd000 bytes) 000444[1] 1788870363.428126 punch from 0x111f1b to 0x11d303 (0xb3e9 bytes) 000447[1] 1788870363.454283 read_OD 0x165000 thru 0x16efff (0xa000 bytes) 000450[0] 1788870363.487872 mapwrite 0x4b0000 thru 0x4b1f0a (0x1f0b bytes) 000451[0] 1788870363.608995 read_OD 0x766000 thru 0x766fff (0x1000 bytes) 000452[0] 1788870363.632996 mapread 0x353000 thru 0x3554ec (0x24ed bytes) 000455[1] 1788870363.676435 read_OD 0x87c000 thru 0x886fff (0xb000 bytes) 000456[1] 1788870363.690258 mapwrite 0x36f000 thru 0x37c824 (0xd825 bytes) 000457[0] 1788870363.874901 read_OD 0x3cb000 thru 0x3d2fff (0x8000 bytes) 000459[1] 1788870363.926540 punch from 0x447164 to 0x454f3d (0xddda bytes) 000461[1] 1788870363.967521 mapwrite 0x557000 thru 0x55c048 (0x5049 bytes) 000462[1] 1788870364.148650 mapread 0x308000 thru 0x31470e (0xc70f bytes) 000464[0] 1788870364.210054 write_OD 0x369000 thru 0x376fff (0xe000 bytes) 000465[0] 1788870364.309682 mapread 0x0d0000 thru 0x0d18a4 (0x18a5 bytes) 000466[1] 1788870364.355331 mapread 0x93e000 thru 0x94dcf4 (0xfcf5 bytes) 000467[1] 1788870364.499714 trunc from 0x9e4626 to 0x746344 (0x29e2e1 bytes) 000470[0] 1788870364.635039 mapwrite 0x46b000 thru 0x46b437 (0x438 bytes) 000472[0] 1788870364.814672 trunc from 0x746345 to 0x69a8e0 (0xaba64 bytes) 000475[1] 1788870364.941258 mapread 0x120000 thru 0x1282bd (0x82be bytes) 000478[0] 1788870365.026120 trunc from 0x69a8e1 to 0x37fd94 (0x31ab4c bytes) 000480[0] 1788870365.086052 punch from 0x0a81d9 to 0x0a9741 (0x1569 bytes) 000481[1] 1788870365.099129 trunc from 0x37fd95 to 0x5c86ee (0x24895a bytes) 000483[1] 1788870365.147864 write_OD 0x5c9000 thru 0x5d7fff (0xf000 bytes) 000484[0] 1788870365.203807 mapread 0x161000 thru 0x1696f9 (0x86fa bytes) 000486[1] 1788870365.256035 trunc from 0x5d8000 to 0x9217de (0x3497df bytes) 000487[0] 1788870365.320013 trunc from 0x9217df to 0x34a1db (0x5d7603 bytes) 000490[1] 1788870365.369576 read_OD 0x1ae000 thru 0x1b5fff (0x8000 bytes) 000491[0] 1788870365.401794 mapwrite 0x81b000 thru 0x8247e3 (0x97e4 bytes) 000492[1] 1788870365.599835 trunc from 0x8247e4 to 0x5536b0 (0x2d1133 bytes) 000495[0] 1788870365.691447 trunc from 0x5536b1 to 0x557add (0x442d bytes) 000497[0] 1788870365.738949 trunc from 0x557ade to 0x6f43ca (0x19c8ed bytes) 000500[1] 1788870365.775881 mapwrite 0x821000 thru 0x830ead (0xfeae bytes) All operations completed A-OK! PASS 16a (61s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 08:26:11 (1788870371) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.511014 s, 16.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.349281 s, 24.0 MB/s Chance of close/open is 1 in 50 Seed set to 374 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788870375.877757 write 0x4d2413 thru 0x4d251c (0x10a bytes) 000002[1] 1788870375.925871 mapread 0x41820d thru 0x419633 (0x1427 bytes) 000004[1] 1788870375.942708 mapwrite 0x2efe8d thru 0x2f17d3 (0x1947 bytes) 000005[0] 1788870376.007808 write 0x84e5e8 thru 0x84f799 (0x11b2 bytes) 000008[1] 1788870376.066553 punch from 0x0ac30e to 0x0ad491 (0x1184 bytes) 000010[1] 1788870376.139455 mapread 0x50fed3 thru 0x511bca (0x1cf8 bytes) 000012[0] 1788870376.175235 punch from 0x7147bd to 0x71640b (0x1c4f bytes) 000014[0] 1788870376.228301 punch from 0x417104 to 0x4174e3 (0x3e0 bytes) 000015[1] 1788870376.251149 read 0x41edf7 thru 0x420d74 (0x1f7e bytes) 000017[1] 1788870376.279683 read 0x3ac61b thru 0x3ad9b2 (0x1398 bytes) 000018[0] 1788870376.286106 mapread 0x1b7c64 thru 0x1b879d (0xb3a bytes) 000019[0] 1788870376.323203 punch from 0x5b7c03 to 0x5b90b8 (0x14b6 bytes) truncating to largest ever: 0x3cb425 000020[1] 1788870376.352317 trunc from 0x84f79a to 0x3cb424 (0x484375 bytes) 000021[1] 1788870376.394464 punch from 0x383da9 to 0x384fe2 (0x123a bytes) truncating to largest ever: 0x88b5b6 000024[0] 1788870376.475804 trunc from 0x3cb425 to 0x88b5b5 (0x4c0191 bytes) 000025[1] 1788870376.559808 mapread 0x1c918f thru 0x1c9b25 (0x997 bytes) 000027[1] 1788870376.622180 read 0x638f38 thru 0x639c39 (0xd02 bytes) 000029[0] 1788870376.692094 mapwrite 0x51e12d thru 0x51e59f (0x473 bytes) 000030[1] 1788870376.889818 write 0x1fd7db thru 0x1fed36 (0x155c bytes) 000031[0] 1788870376.936740 mapread 0x4d64ac thru 0x4d7303 (0xe58 bytes) 000032[0] 1788870376.988572 mapwrite 0x5fbd6a thru 0x5fd685 (0x191c bytes) 000033[0] 1788870377.099147 write 0x014bb8 thru 0x016b2e (0x1f77 bytes) 000036[1] 1788870377.153524 read 0x6579d4 thru 0x6580a8 (0x6d5 bytes) 000037[0] 1788870377.197476 trunc from 0x88b5b6 to 0x44bf0b (0x43f6aa bytes) 000038[0] 1788870377.246689 read 0x338c12 thru 0x3395e7 (0x9d6 bytes) 000039[1] 1788870377.301111 write 0x895878 thru 0x897717 (0x1ea0 bytes) 000040[0] 1788870377.352486 trunc from 0x897718 to 0x3e5803 (0x4b1f14 bytes) 000042[1] 1788870377.506290 mapread 0x211a8b thru 0x212c89 (0x11ff bytes) 000043[0] 1788870377.634481 mapread 0x37a6fb thru 0x37c6d1 (0x1fd7 bytes) 000044[0] 1788870377.722642 mapwrite 0x3d5f31 thru 0x3d7d80 (0x1e50 bytes) 000045[1] 1788870377.845640 punch from 0x2dba2e to 0x2dbc3d (0x210 bytes) 000046[0] 1788870377.918794 punch from 0x2cad44 to 0x2ccc0c (0x1ec9 bytes) 000047[0] 1788870377.962743 mapread 0x231c07 thru 0x2322d4 (0x6ce bytes) 000048[1] 1788870378.079062 write 0x6c7565 thru 0x6c89f8 (0x1494 bytes) 000050[1] 1788870378.143135 read 0x22cd8c thru 0x22e840 (0x1ab5 bytes) 000051[1] 1788870378.159661 write 0x355842 thru 0x357639 (0x1df8 bytes) 000052[1] 1788870378.317659 read 0x6c47fb thru 0x6c548b (0xc91 bytes) 000054[0] 1788870378.356145 mapwrite 0x7c1a84 thru 0x7c2284 (0x801 bytes) 000055[1] 1788870378.609910 mapread 0x3f6534 thru 0x3f8447 (0x1f14 bytes) 000056[0] 1788870378.689638 punch from 0x6b788c to 0x6b8035 (0x7aa bytes) 000057[1] 1788870378.735028 mapwrite 0x268a20 thru 0x268e59 (0x43a bytes) 000058[0] 1788870378.829270 punch from 0x00a9e6 to 0x00c24d (0x1868 bytes) 000061[0] 1788870378.875735 trunc from 0x7c2285 to 0x3796f8 (0x448b8c bytes) 000062[1] 1788870378.917568 mapread 0x03a592 thru 0x03b3f1 (0xe60 bytes) 000064[1] 1788870379.086402 punch from 0x03d7ba to 0x03e6c2 (0xf09 bytes) 000065[0] 1788870379.145976 mapread 0x1807b4 thru 0x181a11 (0x125e bytes) 000066[0] 1788870379.215190 punch from 0x25a17d to 0x25ab5a (0x9de bytes) 000067[1] 1788870379.250408 mapwrite 0x7cfba7 thru 0x7d046d (0x8c7 bytes) 000068[0] 1788870379.437426 mapread 0x581442 thru 0x582f7b (0x1b3a bytes) 000069[0] 1788870379.483330 read 0x1c0620 thru 0x1c124a (0xc2b bytes) 000071[0] 1788870379.517240 mapwrite 0x0fa8fb thru 0x0fc7f9 (0x1eff bytes) 000072[1] 1788870379.617251 read 0x531fc3 thru 0x532d87 (0xdc5 bytes) 000073[1] 1788870379.632975 trunc from 0x7d046e to 0x41f1e1 (0x3b128c bytes) 000074[0] 1788870379.681233 read 0x3464f5 thru 0x3482a0 (0x1dac bytes) 000076[0] 1788870379.723216 punch from 0x226a63 to 0x2273bd (0x95b bytes) 000079[1] 1788870379.787300 write 0x9c58a5 thru 0x9c62eb (0xa47 bytes) 000080[0] 1788870379.799753 mapwrite 0x9ee0f6 thru 0x9ee3bd (0x2c8 bytes) 000081[1] 1788870379.938575 mapwrite 0x8a82a0 thru 0x8a9033 (0xd94 bytes) 000082[1] 1788870380.034434 punch from 0x0f49a9 to 0x0f64f3 (0x1b4b bytes) 000084[0] 1788870380.070029 mapwrite 0x86cce4 thru 0x86e524 (0x1841 bytes) 000088[0] 1788870380.231878 mapread 0x5cd4da thru 0x5cf315 (0x1e3c bytes) 000090[0] 1788870380.257181 read 0x6716ea thru 0x672f49 (0x1860 bytes) 000091[1] 1788870380.264088 punch from 0x5b3dc2 to 0x5b46bc (0x8fb bytes) 000092[1] 1788870380.303545 punch from 0x3fbf8c to 0x3fdac0 (0x1b35 bytes) 000093[0] 1788870380.343933 write 0x914afa thru 0x9158aa (0xdb1 bytes) 000095[1] 1788870380.411554 write 0x531e71 thru 0x533d19 (0x1ea9 bytes) 000096[1] 1788870380.451315 trunc from 0x9ee3be to 0x3e8c4d (0x605770 bytes) 000097[0] 1788870380.513236 mapwrite 0x1f5cf3 thru 0x1f5de5 (0xf3 bytes) 000098[0] 1788870380.607971 write 0x889358 thru 0x8899fd (0x6a6 bytes) 000101[0] 1788870380.682982 mapread 0x59ae96 thru 0x59c837 (0x19a2 bytes) 000102[0] 1788870380.725454 punch from 0x3538df to 0x355821 (0x1f43 bytes) 000103[1] 1788870380.752442 mapwrite 0x961abf thru 0x961f21 (0x463 bytes) 000105[0] 1788870380.851868 punch from 0x3c4c93 to 0x3c641c (0x178a bytes) 000106[0] 1788870380.875842 punch from 0x2a5241 to 0x2a7240 (0x2000 bytes) 000107[0] 1788870380.904924 punch from 0x82ad69 to 0x82c7cc (0x1a64 bytes) 000108[0] 1788870380.924148 write 0x00b47a thru 0x00b77d (0x304 bytes) 000109[0] 1788870380.955374 punch from 0x47e70e to 0x47fb41 (0x1434 bytes) 000110[0] 1788870380.971341 mapread 0x052319 thru 0x053df5 (0x1add bytes) 000112[1] 1788870380.998677 read 0x71e640 thru 0x71fec3 (0x1884 bytes) 000113[1] 1788870381.034146 mapread 0x786783 thru 0x7872d2 (0xb50 bytes) 000114[1] 1788870381.097802 mapwrite 0x138870 thru 0x1392c6 (0xa57 bytes) 000115[0] 1788870381.215688 punch from 0x805a21 to 0x80684e (0xe2e bytes) 000117[1] 1788870381.259271 read 0x6b358b thru 0x6b3e2c (0x8a2 bytes) 000120[0] 1788870381.307827 mapwrite 0x0aa371 thru 0x0abb12 (0x17a2 bytes) 000121[1] 1788870381.435206 punch from 0x22198e to 0x223436 (0x1aa9 bytes) 000122[0] 1788870381.469063 mapwrite 0x3e379b thru 0x3e4908 (0x116e bytes) 000123[0] 1788870381.597561 mapwrite 0x6cb219 thru 0x6cc2cd (0x10b5 bytes) 000125[1] 1788870381.807496 read 0x58d5dc thru 0x58e496 (0xebb bytes) 000127[1] 1788870381.854907 mapwrite 0x54b50c thru 0x54be5f (0x954 bytes) 000128[0] 1788870382.030259 trunc from 0x961f22 to 0x7fb261 (0x166cc0 bytes) 000129[1] 1788870382.086549 trunc from 0x7fb262 to 0x04331c (0x7b7f45 bytes) 000132[0] 1788870382.212149 write 0x66764e thru 0x66828d (0xc40 bytes) 000135[0] 1788870382.320940 punch from 0x4659cd to 0x465e41 (0x475 bytes) 000136[0] 1788870382.347957 trunc from 0x66828e to 0x546694 (0x121bf9 bytes) 000139[0] 1788870382.442263 mapread 0x1a9948 thru 0x1ab1e5 (0x189e bytes) 000140[1] 1788870382.497005 punch from 0x0c29e4 to 0x0c3c33 (0x1250 bytes) 000141[0] 1788870382.540078 trunc from 0x546695 to 0x4e693d (0x5fd57 bytes) 000142[0] 1788870382.579667 punch from 0x4323f1 to 0x432d2a (0x93a bytes) 000143[0] 1788870382.617997 punch from 0x2097ae to 0x20b38c (0x1bdf bytes) 000145[0] 1788870382.669110 trunc from 0x4e693e to 0x3cb39d (0x11b5a0 bytes) 000146[0] 1788870382.689023 punch from 0x1e0dbb to 0x1e2d34 (0x1f7a bytes) 000147[1] 1788870382.693898 mapread 0x37ab26 thru 0x37caad (0x1f88 bytes) 000148[1] 1788870382.751246 read 0x0b6786 thru 0x0b71f6 (0xa71 bytes) 000155[1] 1788870382.880099 mapread 0x0e415e thru 0x0e4a29 (0x8cc bytes) 000157[1] 1788870382.901877 trunc from 0x3cb39e to 0x41c8f3 (0x51556 bytes) 000161[1] 1788870383.059151 trunc from 0x41c8f4 to 0x04edc3 (0x3cdb30 bytes) 000162[1] 1788870383.099415 trunc from 0x04edc4 to 0x27f9fa (0x230c37 bytes) 000164[1] 1788870383.142445 punch from 0x0fccbe to 0x0fea74 (0x1db7 bytes) 000165[1] 1788870383.167367 trunc from 0x27f9fb to 0x85d889 (0x5dde8f bytes) 000166[0] 1788870383.216073 mapwrite 0x7e8ed5 thru 0x7e97c1 (0x8ed bytes) 000167[0] 1788870383.320220 mapread 0x540997 thru 0x5409d6 (0x40 bytes) 000168[0] 1788870383.354029 mapwrite 0x23d5dd thru 0x23dbdb (0x5ff bytes) 000174[0] 1788870383.548350 write 0x9e8173 thru 0x9e9c25 (0x1ab3 bytes) 000175[0] 1788870383.550592 punch from 0x34b17f to 0x34ceb5 (0x1d37 bytes) 000176[1] 1788870383.596357 punch from 0x80e199 to 0x80edc2 (0xc2a bytes) 000177[0] 1788870383.665135 write 0x873a90 thru 0x8742d9 (0x84a bytes) 000178[0] 1788870383.731054 mapread 0x92d2b0 thru 0x92da39 (0x78a bytes) 000179[0] 1788870383.768478 write 0x221b94 thru 0x222448 (0x8b5 bytes) 000185[1] 1788870383.978981 write 0x97f9dc thru 0x981929 (0x1f4e bytes) 000186[0] 1788870384.065874 write 0x396e58 thru 0x3976ef (0x898 bytes) 000188[1] 1788870384.149076 write 0x2e4182 thru 0x2e52dd (0x115c bytes) 000190[0] 1788870384.292592 mapwrite 0x79c64d thru 0x79cffa (0x9ae bytes) 000198[0] 1788870384.550354 read 0x4d2b88 thru 0x4d4897 (0x1d10 bytes) 000199[0] 1788870384.580030 mapwrite 0x86f85b thru 0x870afd (0x12a3 bytes) 000200[1] 1788870384.709211 trunc from 0x9e9c26 to 0x162560 (0x8876c5 bytes) 000205[0] 1788870384.799298 read 0x02089f thru 0x02251f (0x1c81 bytes) 000213[0] 1788870384.959477 read 0x0cc438 thru 0x0ce10a (0x1cd3 bytes) 000215[1] 1788870384.960492 read 0x0567da thru 0x057b09 (0x1330 bytes) 000217[1] 1788870385.030327 write 0x9a05ee thru 0x9a1e0d (0x1820 bytes) 000218[0] 1788870385.090224 mapread 0x98825f thru 0x989c78 (0x1a1a bytes) 000219[1] 1788870385.226113 mapwrite 0x79946a thru 0x79b0e5 (0x1c7c bytes) 000220[1] 1788870385.472064 write 0x0dbad1 thru 0x0dbb11 (0x41 bytes) 000221[1] 1788870385.554429 mapwrite 0x0bce77 thru 0x0bcf3c (0xc6 bytes) 000222[0] 1788870385.815432 mapwrite 0x6db3b6 thru 0x6dc83d (0x1488 bytes) 000224[1] 1788870386.053343 mapwrite 0x5f4fc7 thru 0x5f63fe (0x1438 bytes) 000225[0] 1788870386.177798 read 0x36c738 thru 0x36e0df (0x19a8 bytes) 000226[0] 1788870386.210287 write 0x85ad75 thru 0x85c1ac (0x1438 bytes) 000227[0] 1788870386.264738 write 0x797475 thru 0x798135 (0xcc1 bytes) 000228[0] 1788870386.384116 punch from 0x484fa9 to 0x4860eb (0x1143 bytes) 000229[0] 1788870386.392312 trunc from 0x9a1e0e to 0x2ae8c0 (0x6f354d bytes) 000230[0] 1788870386.418649 punch from 0x26e7e4 to 0x26f92d (0x114a bytes) 000232[0] 1788870386.491880 mapwrite 0x7baddc thru 0x7bc421 (0x1646 bytes) 000233[1] 1788870386.629784 mapread 0x76306c thru 0x763592 (0x527 bytes) 000234[0] 1788870386.717390 write 0x21470c thru 0x215ee5 (0x17da bytes) 000235[0] 1788870386.742422 mapread 0x02faa8 thru 0x031715 (0x1c6e bytes) 000238[1] 1788870386.765371 punch from 0x7b0b66 to 0x7b1bc1 (0x105c bytes) 000240[0] 1788870386.842524 punch from 0x5f6298 to 0x5f8123 (0x1e8c bytes) 000241[1] 1788870386.864130 write 0x2a66d7 thru 0x2a7173 (0xa9d bytes) 000242[1] 1788870386.928729 trunc from 0x7bc422 to 0x374a2d (0x4479f4 bytes) truncating to largest ever: 0x8b8ada 000243[0] 1788870387.030925 trunc from 0x374a2e to 0x8b8ad9 (0x5440ac bytes) 000245[1] 1788870387.101188 punch from 0x4eab96 to 0x4eb91e (0xd89 bytes) 000246[0] 1788870387.141101 read 0x4bb72d thru 0x4bbb18 (0x3ec bytes) 000249[1] 1788870387.166327 mapread 0x1a9288 thru 0x1aadd1 (0x1b4a bytes) 000250[0] 1788870387.236634 read 0x84fbf1 thru 0x850fa3 (0x13b3 bytes) 000251[0] 1788870387.257120 mapread 0x636390 thru 0x636612 (0x283 bytes) 000252[1] 1788870387.273094 punch from 0x05bbd2 to 0x05c692 (0xac1 bytes) 000253[1] 1788870387.349520 read 0x3dab7f thru 0x3dc069 (0x14eb bytes) 000254[1] 1788870387.404252 mapread 0x2af564 thru 0x2b05d2 (0x106f bytes) 000255[1] 1788870387.443220 trunc from 0x8b8ada to 0x4be7e8 (0x3fa2f1 bytes) 000256[1] 1788870387.613644 read 0x2ba23a thru 0x2bb323 (0x10ea bytes) 000257[0] 1788870387.630279 read 0x22890b thru 0x229bc3 (0x12b9 bytes) 000258[0] 1788870387.735355 trunc from 0x4be7e9 to 0x726c00 (0x268418 bytes) 000259[1] 1788870387.897982 trunc from 0x726c01 to 0x09d3bc (0x689844 bytes) 000260[0] 1788870387.974162 mapwrite 0x2f6295 thru 0x2f76e9 (0x1455 bytes) 000261[0] 1788870388.128370 mapread 0x080d5f thru 0x081a1f (0xcc1 bytes) 000262[1] 1788870388.139797 punch from 0x2f31a3 to 0x2f4a9c (0x18fa bytes) 000265[1] 1788870388.257176 read 0x239f84 thru 0x23aaa8 (0xb25 bytes) 000266[0] 1788870388.443479 write 0x7ca56e thru 0x7ca7ff (0x292 bytes) 000267[1] 1788870388.478564 trunc from 0x7ca800 to 0x39df40 (0x42c8bf bytes) 000269[0] 1788870388.528137 trunc from 0x39df41 to 0x153ae1 (0x24a45f bytes) 000270[1] 1788870388.582566 punch from 0x07b238 to 0x07b616 (0x3df bytes) 000273[0] 1788870388.649949 write 0x740f83 thru 0x74228a (0x1308 bytes) 000274[1] 1788870388.671833 read 0x338d08 thru 0x33a005 (0x12fe bytes) 000275[0] 1788870388.793267 punch from 0x4474d7 to 0x4492c2 (0x1dec bytes) 000276[0] 1788870388.817494 write 0x4246d2 thru 0x425987 (0x12b6 bytes) 000277[0] 1788870388.846686 trunc from 0x74228b to 0x60fa3f (0x13284b bytes) 000278[1] 1788870388.912857 write 0x82dfdb thru 0x82f838 (0x185e bytes) 000279[0] 1788870388.950440 mapwrite 0x6606b3 thru 0x6613cb (0xd19 bytes) 000282[0] 1788870389.314172 trunc from 0x82f839 to 0x712daa (0x11ca8e bytes) 000283[0] 1788870389.371091 trunc from 0x712dab to 0x552672 (0x1c0738 bytes) 000285[0] 1788870389.435903 mapread 0x385fba thru 0x387fad (0x1ff4 bytes) 000286[0] 1788870389.483558 mapwrite 0x96e937 thru 0x96ed8c (0x456 bytes) 000289[0] 1788870389.726432 mapwrite 0x623dee thru 0x625470 (0x1683 bytes) truncating to largest ever: 0x973862 000291[1] 1788870389.902748 trunc from 0x96ed8d to 0x973861 (0x4ad5 bytes) 000292[0] 1788870390.053324 mapwrite 0x9b198a thru 0x9b1fde (0x655 bytes) 000294[1] 1788870390.334637 trunc from 0x9b1fdf to 0x095a4d (0x91c591 bytes) 000297[1] 1788870390.493019 mapwrite 0x8ab30b thru 0x8acee9 (0x1bdf bytes) 000299[0] 1788870390.960336 read 0x48e224 thru 0x48f917 (0x16f4 bytes) 000301[1] 1788870391.019974 punch from 0x29e481 to 0x29e510 (0x90 bytes) 000304[0] 1788870391.049018 mapread 0x826a3e thru 0x828505 (0x1ac8 bytes) 000306[1] 1788870391.116187 mapread 0x707af5 thru 0x708395 (0x8a1 bytes) 000307[1] 1788870391.133763 mapread 0x216395 thru 0x216cdb (0x947 bytes) 000308[1] 1788870391.143109 mapread 0x4bd3f0 thru 0x4bdfb8 (0xbc9 bytes) 000309[0] 1788870391.158247 mapread 0x23eb56 thru 0x23f100 (0x5ab bytes) 000310[0] 1788870391.187165 punch from 0x079d4c to 0x07b768 (0x1a1d bytes) 000312[1] 1788870391.326462 write 0x0ea908 thru 0x0eb417 (0xb10 bytes) 000313[1] 1788870391.378362 write 0x6f1557 thru 0x6f21cc (0xc76 bytes) 000314[1] 1788870391.504526 read 0x23a80f thru 0x23ab39 (0x32b bytes) 000315[0] 1788870391.558710 punch from 0x6d7184 to 0x6d76ec (0x569 bytes) 000319[1] 1788870391.728544 trunc from 0x8aceea to 0x31f5a3 (0x58d946 bytes) 000321[0] 1788870391.838394 trunc from 0x31f5a4 to 0x82a71c (0x50b179 bytes) 000322[1] 1788870391.974979 mapread 0x78a226 thru 0x78befa (0x1cd5 bytes) 000323[0] 1788870392.034937 trunc from 0x82a71d to 0x8484eb (0x1ddcf bytes) 000324[1] 1788870392.115245 mapread 0x662df9 thru 0x66365b (0x863 bytes) 000327[1] 1788870392.220320 write 0x0cb783 thru 0x0cca5a (0x12d8 bytes) 000329[0] 1788870392.268986 write 0x71ad7f thru 0x71be51 (0x10d3 bytes) 000330[0] 1788870392.315076 mapwrite 0x7f820c thru 0x7f9d2b (0x1b20 bytes) 000331[0] 1788870392.468169 read 0x6b48bc thru 0x6b492e (0x73 bytes) 000335[0] 1788870392.579020 read 0x3adcca thru 0x3af1a6 (0x14dd bytes) 000336[1] 1788870392.650038 trunc from 0x8484ec to 0x2f1b69 (0x556982 bytes) 000337[0] 1788870392.740583 read 0x201090 thru 0x20146b (0x3dc bytes) 000339[1] 1788870392.868642 punch from 0x27346c to 0x273b02 (0x697 bytes) 000340[1] 1788870392.928748 trunc from 0x2f1b6a to 0x01cd4b (0x2d4e1e bytes) 000342[1] 1788870393.028768 trunc from 0x01cd4c to 0x2bef39 (0x2a21ee bytes) 000343[0] 1788870393.105050 mapwrite 0x87025f thru 0x8706a0 (0x442 bytes) 000345[0] 1788870393.353431 punch from 0x61edc2 to 0x620724 (0x1963 bytes) 000346[0] 1788870393.416275 mapread 0x6e9f5f thru 0x6eb7da (0x187c bytes) 000349[1] 1788870393.449321 punch from 0x72f3ad to 0x72f465 (0xb9 bytes) 000352[1] 1788870393.584324 mapread 0x2dbd31 thru 0x2dd67c (0x194c bytes) 000354[0] 1788870393.679519 mapwrite 0x9e128e thru 0x9e2f48 (0x1cbb bytes) 000357[0] 1788870394.135957 read 0x5db383 thru 0x5dc8a1 (0x151f bytes) 000359[0] 1788870394.147849 read 0x79f5a3 thru 0x79fbba (0x618 bytes) 000360[0] 1788870394.181145 trunc from 0x9e2f49 to 0x17cf84 (0x865fc4 bytes) 000361[1] 1788870394.318374 trunc from 0x17cf85 to 0x37ba1d (0x1fea99 bytes) 000363[1] 1788870394.486889 punch from 0x327805 to 0x3281c1 (0x9bd bytes) 000364[1] 1788870394.543723 read 0x1b5480 thru 0x1b72e2 (0x1e63 bytes) 000365[0] 1788870394.662210 write 0x66c9cd thru 0x66cc9c (0x2d0 bytes) 000366[1] 1788870394.697685 mapread 0x4a7046 thru 0x4a7b38 (0xaf3 bytes) 000369[1] 1788870394.756746 read 0x581e06 thru 0x583082 (0x127d bytes) 000370[0] 1788870394.777971 write 0x90f0c8 thru 0x910526 (0x145f bytes) 000371[0] 1788870394.821570 write 0x679134 thru 0x6791de (0xab bytes) truncating to largest ever: 0x9d9a68 000372[0] 1788870394.896839 trunc from 0x910527 to 0x9d9a67 (0xc9541 bytes) 000374[1] 1788870395.023576 mapread 0x3dc151 thru 0x3dc8ec (0x79c bytes) 000375[0] 1788870395.129139 mapread 0x3f8a41 thru 0x3f9801 (0xdc1 bytes) 000377[1] 1788870395.187964 write 0x11acee thru 0x11b57d (0x890 bytes) 000379[1] 1788870395.360468 read 0x606dc3 thru 0x6071da (0x418 bytes) 000382[0] 1788870395.494419 trunc from 0x9d9a68 to 0x201fcf (0x7d7a98 bytes) 000384[0] 1788870395.663379 punch from 0x0cc4f3 to 0x0ce18f (0x1c9d bytes) 000385[1] 1788870395.671592 trunc from 0x201fd0 to 0x0bb118 (0x146eb7 bytes) 000387[1] 1788870395.793452 mapread 0x00d785 thru 0x00db2c (0x3a8 bytes) 000388[1] 1788870395.852512 write 0x540de3 thru 0x541c95 (0xeb3 bytes) 000391[1] 1788870395.923659 punch from 0x07d8ea to 0x07eb94 (0x12ab bytes) 000392[1] 1788870395.936415 trunc from 0x541c96 to 0x8ceb62 (0x38cecd bytes) 000393[1] 1788870396.013321 trunc from 0x8ceb63 to 0x071d01 (0x85ce61 bytes) 000396[0] 1788870396.114932 trunc from 0x071d02 to 0x146063 (0xd4362 bytes) 000400[1] 1788870396.237451 read 0x1039aa thru 0x10458d (0xbe4 bytes) 000403[1] 1788870396.387291 punch from 0x089d54 to 0x08afb8 (0x1265 bytes) 000405[0] 1788870396.471985 punch from 0x028520 to 0x029076 (0xb57 bytes) 000407[1] 1788870396.590510 mapwrite 0x566267 thru 0x566880 (0x61a bytes) 000408[0] 1788870396.977110 mapwrite 0x4da812 thru 0x4dbeab (0x169a bytes) 000409[0] 1788870397.230253 mapwrite 0x5b861d thru 0x5b8eea (0x8ce bytes) 000412[0] 1788870397.463432 punch from 0x491d2b to 0x49210d (0x3e3 bytes) 000414[0] 1788870397.570745 mapwrite 0x0c7d2c thru 0x0c94e1 (0x17b6 bytes) 000416[0] 1788870397.836475 trunc from 0x5b8eeb to 0x29f978 (0x319572 bytes) 000417[1] 1788870397.880660 mapwrite 0x47eed2 thru 0x4809dd (0x1b0c bytes) 000418[1] 1788870398.108400 write 0x9ea977 thru 0x9eab69 (0x1f3 bytes) 000420[0] 1788870398.158427 read 0x845dbb thru 0x846322 (0x568 bytes) 000421[0] 1788870398.200009 mapwrite 0x766aeb thru 0x767a38 (0xf4e bytes) 000422[1] 1788870398.353901 read 0x83fe96 thru 0x8407b7 (0x922 bytes) 000423[1] 1788870398.378030 punch from 0x8bfde4 to 0x8c110b (0x1328 bytes) 000424[0] 1788870398.428583 read 0x4dfc8f thru 0x4e1adb (0x1e4d bytes) 000426[1] 1788870398.509413 trunc from 0x9eab6a to 0x51266d (0x4d84fc bytes) 000428[1] 1788870398.603416 punch from 0x2c341f to 0x2c409f (0xc81 bytes) 000429[0] 1788870398.623441 trunc from 0x51266e to 0x64c1e1 (0x139b74 bytes) 000430[0] 1788870398.792645 mapread 0x5a6c7a thru 0x5a6d25 (0xac bytes) 000432[0] 1788870398.893663 mapwrite 0x180407 thru 0x1823b1 (0x1fab bytes) 000434[0] 1788870399.143147 mapread 0x51260f thru 0x51459e (0x1f90 bytes) 000436[1] 1788870399.191541 punch from 0x60de2a to 0x60e6d4 (0x8ab bytes) 000437[0] 1788870399.274261 mapwrite 0x9f34b3 thru 0x9f493b (0x1489 bytes) 000438[0] 1788870399.425792 write 0x3caa8e thru 0x3cab5b (0xce bytes) 000439[0] 1788870399.434926 punch from 0x765b09 to 0x767608 (0x1b00 bytes) 000442[0] 1788870399.454464 write 0x89ef27 thru 0x8a03be (0x1498 bytes) 000443[1] 1788870399.487163 trunc from 0x9f493c to 0x8d8696 (0x11c2a5 bytes) 000446[1] 1788870399.617311 punch from 0x12c762 to 0x12dc2e (0x14cd bytes) 000447[0] 1788870399.646305 write 0x0b064e thru 0x0b1864 (0x1217 bytes) 000448[1] 1788870399.696989 trunc from 0x8d8697 to 0x6ba6b8 (0x21dfde bytes) 000449[1] 1788870399.748812 mapwrite 0x90f533 thru 0x91147c (0x1f4a bytes) 000450[0] 1788870399.911426 mapread 0x40d43e thru 0x40d631 (0x1f4 bytes) 000452[1] 1788870400.038774 mapwrite 0x913407 thru 0x914212 (0xe0c bytes) 000454[1] 1788870400.217105 read 0x78a397 thru 0x78a64e (0x2b8 bytes) 000456[1] 1788870400.342736 trunc from 0x914213 to 0x6b88aa (0x25b968 bytes) 000457[1] 1788870400.436930 mapwrite 0x49532f thru 0x495c57 (0x929 bytes) 000458[1] 1788870400.552383 mapread 0x54297e thru 0x542f28 (0x5ab bytes) 000460[0] 1788870400.607732 write 0x64ae32 thru 0x64b75f (0x92e bytes) 000464[1] 1788870400.848316 write 0x997f42 thru 0x999542 (0x1601 bytes) 000465[1] 1788870400.889741 mapread 0x4be265 thru 0x4be972 (0x70e bytes) 000466[0] 1788870400.950877 read 0x80eb00 thru 0x80f5e5 (0xae6 bytes) 000468[1] 1788870401.020024 mapread 0x8bb677 thru 0x8bb705 (0x8f bytes) 000469[0] 1788870401.028040 read 0x9193cd thru 0x91a4a7 (0x10db bytes) 000470[0] 1788870401.041074 mapwrite 0x37c36c thru 0x37d1af (0xe44 bytes) 000472[0] 1788870401.329451 mapwrite 0x2168ce thru 0x21746d (0xba0 bytes) 000476[0] 1788870401.477535 mapwrite 0x0edd95 thru 0x0ee73d (0x9a9 bytes) 000478[1] 1788870401.604093 write 0x3a62cd thru 0x3a67b1 (0x4e5 bytes) 000479[0] 1788870401.649956 mapwrite 0x7b90b3 thru 0x7b9909 (0x857 bytes) 000480[0] 1788870401.919406 mapwrite 0x2c7d83 thru 0x2c8aa5 (0xd23 bytes) 000481[0] 1788870402.167294 mapread 0x2ed7f8 thru 0x2eee13 (0x161c bytes) 000482[0] 1788870402.193659 mapread 0x2dfc3d thru 0x2e16bd (0x1a81 bytes) 000483[0] 1788870402.209721 trunc from 0x999543 to 0x16e506 (0x82b03c bytes) 000486[0] 1788870402.391245 punch from 0x0fcf67 to 0x0fe744 (0x17de bytes) 000488[0] 1788870402.434215 mapwrite 0x7f849f thru 0x7fa365 (0x1ec7 bytes) 000490[1] 1788870402.707176 mapwrite 0x118e0e thru 0x11a34c (0x153f bytes) 000492[1] 1788870403.046797 write 0x8ebfb2 thru 0x8ed5f0 (0x163f bytes) 000493[0] 1788870403.072520 trunc from 0x8ed5f1 to 0x2b338d (0x63a263 bytes) 000495[1] 1788870403.187209 read 0x1f5cf3 thru 0x1f5d06 (0x14 bytes) 000496[1] 1788870403.297549 mapwrite 0x1f2b29 thru 0x1f38d2 (0xdaa bytes) 000497[1] 1788870403.594501 write 0x9a5a01 thru 0x9a626f (0x86f bytes) 000498[1] 1788870403.635438 write 0x08b34a thru 0x08b76c (0x423 bytes) 000499[1] 1788870403.723379 mapwrite 0x5b942f thru 0x5ba8d0 (0x14a2 bytes) All operations completed A-OK! PASS 16b (40s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 08:26:51 (1788870411) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (6s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 08:26:57 (1788870417) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.446584 s, 18.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.365129 s, 23.0 MB/s ldlm.namespaces.MGC192.168.202.155@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf40343b000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf405439800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cf40343b000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cf40343b000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.7631 s, 7.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.56459 s, 29.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.55021 s, 41.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.14997 s, 14.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.16381 s, 11.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.35291 s, 24.1 MB/s PASS 16d (58s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 08:27:56 (1788870476) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.403393 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165202 s, 6.3 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 08:28:05 (1788870485) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 08:28:34 (1788870514) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 08:29:03 (1788870543) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 08:29:10 (1788870550) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 08:29:19 (1788870559) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.74071 s, 9.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62392 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75278 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98003 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22597 s, 11.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.97199 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40363 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.72729 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06219 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.47183 s, 7.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.69256 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56761 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.92565 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.79481 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.92639 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04812 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50565 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.80878 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.80064 s, 6.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 08:30:01 (1788870601) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 8 08:30:49 EDT 2026 + fsx 2 done: rc=0 Tue Sep 8 08:30:49 EDT 2026 + fsx 3 done: rc=0 Tue Sep 8 08:30:49 EDT 2026 + fsx 4 done: rc=0 Tue Sep 8 08:30:49 EDT 2026 + fsx 5 done: rc=0 Tue Sep 8 08:30:49 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 08:30:58 (1788870658) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 08:31:09 (1788870669) mmap test1: basic mmap operation (PASS, 0.11632s) mmap test2: MAP_PRIVATE not write back (PASS, 0.047229s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.5192s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5134s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11051s) mmap test6: check mmap write/read content on two nodes (PASS, 0.191s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.132s) mmap test11: multi-thread shared mmap (PASS, 1.2389s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1354s) PASS 18 (23s) == sanityn test 19: test concurrent uncached read races ========================================================================= 08:31:33 (1788870693) oleg255-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 08:31:37 (1788870697) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 08:31:45 (1788870705) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 08:31:53 (1788870713) now is 1788870716 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 new atime is 1788870777 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:33:04 (1788870784) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:33:11 (1788870791) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:33:18 (1788870798) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:33:26 (1788870806) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 08:33:30 (1788870810) utime: good mknod times 1788870811 <= 1788870812 <= 1788870812 for /mnt/lustre/f26a utime: good mknod times 1788870811 <= 1788870812 <= 1788870812 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 08:33:38 (1788870818) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:33:47 (1788870827) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 08:33:49.000000000 -0400 Birth: 2026-09-08 08:33:49.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 08:33:49.000000000 -0400 Birth: 2026-09-08 08:33:49.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:33:57 (1788870837) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:34:07 (1788870847) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:34:17 (1788870857) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.155@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 08:34:42 (1788870882) -rw-r--r-- 1 root root 0 Sep 8 08:34 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:34:53 (1788870893) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43175 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:35:00 (1788870900) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 08:35:07 (1788870907) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 08:35:11 (1788870911) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.83617 s, 28.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36265 s, 38.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.03963 s, 25.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.30527 s, 22.7 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.07505 s, 25.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.24691 s, 23.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71197 s, 30.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71407 s, 30.6 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:35:43 (1788870943) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:35:48 (1788870948) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:35:51 (1788870951) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:35:54 (1788870954) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 08:35:58 (1788870958) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid in FULL state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid in FULL state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid in IDLE state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf40343b000.ost_server_uuid in FULL state after 0 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf40343b000.ost_server_uuid in IDLE state after 0 sec PASS 34 (140s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:38:17 (1788871097) fail_loc=0x80000317 Race attempt 0 Wait for 50481 50509 for 60 sec... - open/close 905 (time 1788871115.05 total 10.01 last 90.44) - open/close 1759 (time 1788871125.05 total 20.01 last 85.39) - open/close 2578 (time 1788871135.06 total 30.02 last 81.83) - open/close 3290 (time 1788871145.07 total 40.02 last 71.15) total: 4000 open/close in 49.12 seconds: 81.44 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.155@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.155@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:39:32 (1788871172) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg255-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29453 s, 40.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.17137 s, 44.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41905 s, 36.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50121 s, 34.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64638 s, 31.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7490560) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6997 s, 30.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33171 s, 39.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61136 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7442432) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26352 s, 41.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7442432) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59481 s, 32.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7442432) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95777 s, 26.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7442432) after(7495680) PASS 36 (348s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 08:45:20 (1788871520) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 - create 1338 (time 1788871535.64 total 10.01 last 133.61) - create 2674 (time 1788871545.66 total 20.03 last 133.41) - create 4127 (time 1788871555.66 total 30.03 last 145.24) - create 5337 (time 1788871565.66 total 40.03 last 121.00) - create 6760 (time 1788871575.67 total 50.04 last 142.23) - create 8241 (time 1788871585.67 total 60.04 last 148.04) - create 9534 (time 1788871595.67 total 70.04 last 129.26) total: 10000 create in 75.14 seconds: 133.09 ops/second PASS 37 (90s) == sanityn test 39a: file mtime does not change after rename ========================================================== 08:46:51 (1788871611) repeat after cancel_lru_locks PASS 39a (10s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 08:47:00 (1788871620) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 08:47:09 (1788871629) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 08:47:17 (1788871637) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 08:47:28 (1788871648) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (21s) == sanityn test 40c: pdirops: link and others ======================================================================== 08:47:49 (1788871669) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 08:48:07 (1788871687) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 08:48:26 (1788871706) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 08:48:43 (1788871723) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 08:48:55 (1788871735) open: File exists Conflict PASS 41b (18s) == sanityn test 41c: pdirops: create vs link ======================================================================== 08:49:13 (1788871753) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (16s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 08:49:29 (1788871769) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 08:49:43 (1788871783) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 08:49:59 (1788871799) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 08:50:15 (1788871815) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 08:50:28 (1788871828) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 08:50:42 (1788871842) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1115s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 09:09:17 (1788872957) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 09:09:31 (1788872971) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 09:09:45 (1788872985) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 09:10:01 (1788873001) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 09:10:16 (1788873016) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 09:10:30 (1788873030) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 09:10:46 (1788873046) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 09:10:59 (1788873059) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 09:11:12 (1788873072) PASS 43a (60s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 09:12:12 (1788873132) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 09:12:26 (1788873146) Conflict PASS 43c (18s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 09:12:44 (1788873164) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 09:12:59 (1788873179) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 09:13:13 (1788873193) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 09:13:27 (1788873207) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 09:13:42 (1788873222) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 09:13:57 (1788873237) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 09:14:01 (1788873241) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (140s) == sanityn test 43k: unlink vs create ==================== 09:16:21 (1788873381) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1118s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 09:34:59 (1788874499) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 8 09:35:09 EDT 2026 PASS 44a (16s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 09:35:15 (1788874515) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 09:35:29 (1788874529) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 09:35:44 (1788874544) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 09:35:57 (1788874557) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 09:36:12 (1788874572) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 09:36:27 (1788874587) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 09:36:41 (1788874601) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 09:36:55 (1788874615) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 09:36:59 (1788874619) PASS 45a (111s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 09:38:50 (1788874730) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 09:39:03 (1788874743) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 09:39:18 (1788874758) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 09:39:34 (1788874774) Conflict PASS 45e (16s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 09:39:50 (1788874790) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (17s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 09:40:07 (1788874807) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 09:40:22 (1788874822) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 09:40:35 (1788874835) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 09:40:38 (1788874838) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1040s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 09:57:58 (1788875878) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 09:58:09 (1788875889) open: File exists Conflict PASS 46b (10s) == sanityn test 46c: pdirops: link vs link ======================================================================== 09:58:20 (1788875900) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (10s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 09:58:30 (1788875910) Conflict PASS 46d (9s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 09:58:39 (1788875919) Conflict PASS 46e (10s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 09:58:49 (1788875929) Conflict PASS 46f (10s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 09:58:59 (1788875939) Conflict PASS 46g (9s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 09:59:08 (1788875948) Conflict PASS 46h (8s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 09:59:17 (1788875957) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 09:59:18 (1788875958) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 09:59:20 (1788875960) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 09:59:22 (1788875962) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 09:59:24 (1788875964) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 09:59:26 (1788875966) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 09:59:28 (1788875968) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 09:59:30 (1788875970) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 09:59:32 (1788875972) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.113865 s, 89.9 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (9s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 09:59:41 (1788875981) 0+1 records in 0+1 records out 158 bytes copied, 0.0591924 s, 2.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218992) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 09:59:48 (1788875988) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00601273 s, 170 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443101 s, 23.7 MB/s 3145728 PASS 51b (17s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:00:05 (1788876005) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0106959 s, 95.7 kB/s PASS 51c (9s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:00:14 (1788876014) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445815 s, 23.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:00:19 (1788876019) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 10:00:26 (1788876026) llite.lustre-ffff9cf40343b000.xattr_cache=1 llite.lustre-ffff9cf405439800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:28.000000000 -0400 Modify: 2026-09-08 10:00:28.000000000 -0400 Change: 2026-09-08 10:00:28.000000000 -0400 Birth: 2026-09-08 10:00:28.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:33.000000000 -0400 Modify: 2026-09-08 10:00:34.000000000 -0400 Change: 2026-09-08 10:00:34.000000000 -0400 Birth: 2026-09-08 10:00:34.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:39.000000000 -0400 Modify: 2026-09-08 10:00:39.000000000 -0400 Change: 2026-09-08 10:00:39.000000000 -0400 Birth: 2026-09-08 10:00:40.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:39.000000000 -0400 Modify: 2026-09-08 10:00:40.000000000 -0400 Change: 2026-09-08 10:00:40.000000000 -0400 Birth: 2026-09-08 10:00:40.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:45.000000000 -0400 Modify: 2026-09-08 10:00:46.000000000 -0400 Change: 2026-09-08 10:00:46.000000000 -0400 Birth: 2026-09-08 10:00:46.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:00:45.000000000 -0400 Modify: 2026-09-08 10:00:46.000000000 -0400 Change: 2026-09-08 10:00:46.000000000 -0400 Birth: 2026-09-08 10:00:46.000000000 -0400 PASS 54 (28s) == sanityn test 55a: rename vs unlink target dir ========= 10:00:54 (1788876054) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 10:01:04 (1788876064) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 10:01:14 (1788876074) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 10:01:31 (1788876091) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:01:44 (1788876104) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:01:46 (1788876106) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (16s) == sanityn test 55g: rename: race with trylock =========== 10:02:02 (1788876122) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (19s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:02:21 (1788876141) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788876142 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.871987 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (75s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 10:03:36 (1788876216) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788876220 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.71936 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (80s) == sanityn test 60: Verify data_version behaviour ======== 10:04:56 (1788876296) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 10:05:02 (1788876302) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 10:05:09 (1788876309) PASS 70b (6s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 10:05:15 (1788876315) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00980352 s, 6.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 10:05:17 (1788876317) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00973524 s, 6.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 10:05:19 (1788876319) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 10:05:22 (1788876322) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0218588 s, 3.0 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 10:05:25 (1788876325) llite.lustre-ffff9cf40343b000.xattr_cache=1 llite.lustre-ffff9cf405439800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (5s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 10:05:30 (1788876330) llite.lustre-ffff9cf40343b000.xattr_cache=1 llite.lustre-ffff9cf405439800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 10:05:35 (1788876335) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00800725 s, 128 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0139887 s, 73.2 kB/s 231787: get lock2 231788: get lock1 231787: done 231787: get lock1 231788: done 231788: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231788: cannot get lock2: Resource deadlock avoided 231788: exit rc=0 231787: done 231787: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 10:05:44 (1788876344) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0513569 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262103 s, 40.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232368 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 10:05:55 (1788876355) opening files: ........ closing files: ........ PASS 76 (51s) == sanityn test 77a: check FIFO NRS policy =============== 10:06:46 (1788876406) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279535 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334502 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682295 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701026 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555962 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123959 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651787 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140262 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116365 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140081 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217419 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181118 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228935 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215277 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208545 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222907 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297078 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388179 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303796 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049071 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585869 s, 17.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518572 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655834 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060297 s, 17.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771448 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802954 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618588 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707437 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672404 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525746 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705967 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632936 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105762 s, 9.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (6s) == sanityn test 77b: check CRR-N NRS policy ============== 10:06:53 (1788876413) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467066 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791973 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799834 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762491 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174358 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174449 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226846 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235994 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246881 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238901 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242017 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229608 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237546 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299319 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277761 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294414 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319123 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730719 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069041 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351374 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569173 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477195 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338317 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411259 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494251 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654092 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569962 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310484 s, 338 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510603 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617796 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382863 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102711 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537206 s, 19.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317174 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618072 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873052 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105352 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201609 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174445 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178028 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212307 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19732 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177857 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232477 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194317 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245025 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218786 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242996 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254445 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210469 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335323 s, 313 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667766 s, 157 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404963 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462589 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953094 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781559 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649808 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952308 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592539 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657034 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198877 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334168 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271182 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331276 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159514 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250711 s, 41.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (10s) == sanityn test 77c: check ORR NRS policy ================ 10:07:03 (1788876423) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.321378 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561732 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864607 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782783 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125281 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196722 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221291 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230669 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262796 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285791 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296869 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299709 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30111 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318566 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314402 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121901 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324532 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216656 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387399 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332546 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117581 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117773 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564215 s, 18.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458971 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124007 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047482 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695213 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705845 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747481 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736651 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760242 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179131 s, 5.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.415501 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334076 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521257 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795539 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501063 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551195 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741003 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069266 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618593 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083093 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101381 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144633 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167024 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171039 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184566 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179577 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227797 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686081 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498442 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753229 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700728 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632733 s, 16.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433367 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440485 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764378 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763262 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717038 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872569 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897162 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891299 s, 11.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912106 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707019 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915071 s, 11.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.342346 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458718 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540259 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139424 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149539 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161635 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186311 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204049 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220695 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208056 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219657 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277581 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242041 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283191 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30711 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30438 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284881 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335247 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200839 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229144 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459035 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421992 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626576 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644378 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397866 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622627 s, 16.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795999 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621766 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670271 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558845 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591687 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732215 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842458 s, 12.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (11s) == sanityn test 77d: check TRR nrs policy ================ 10:07:15 (1788876435) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.367765 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554929 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981279 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058555 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582124 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127806 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141057 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159483 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177751 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15331 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100066 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133712 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192498 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141213 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229607 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21305 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293342 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389434 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074488 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666186 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402633 s, 260 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924639 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750841 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608152 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788899 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659219 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957898 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495641 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722729 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789472 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064199 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626292 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648888 s, 16.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.325957 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109432 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133414 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152769 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106592 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155088 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171258 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164465 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152467 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208643 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20503 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234353 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267551 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266078 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248326 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164564 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271532 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608353 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861742 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313836 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815642 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574466 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740484 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531246 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850422 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458099 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474122 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874382 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892303 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949437 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872187 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130472 s, 80.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557534 s, 18.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.363659 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732219 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823517 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128791 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11762 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159148 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186766 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193514 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217999 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210389 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185685 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169825 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264384 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264437 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213197 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246816 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299221 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398711 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373922 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617101 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072782 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069953 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835371 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459227 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381625 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514176 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472991 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503397 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596127 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513176 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567007 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713426 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633244 s, 16.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (12s) == sanityn test 77e: check TBF NID nrs policy ============ 10:07:27 (1788876447) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.55@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.335169 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399744 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542786 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163269 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161342 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185294 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150544 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190454 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191223 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17368 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182807 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167435 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203365 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879303 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165521 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168873 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221737 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137673 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143338 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126694 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139519 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108376 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111237 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132373 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135085 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157577 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129117 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119851 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122865 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126312 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169918 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12732 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131996 s, 7.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402798 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568393 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971194 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112744 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105012 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115565 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103582 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763503 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162628 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132205 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181865 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160867 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185694 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164077 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218598 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231764 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162197 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203564 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556374 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337394 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550158 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554271 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728885 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352577 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362499 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387103 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876826 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730424 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925047 s, 11.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955223 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102547 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664436 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840278 s, 12.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.330402 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041791 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061815 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990388 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126339 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16738 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191356 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159157 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181432 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196972 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202509 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17352 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187911 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223227 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237866 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237413 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293171 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392275 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958719 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333995 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443194 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00314588 s, 333 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576755 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314235 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524811 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426364 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286151 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018765 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245297 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153023 s, 68.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035356 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296008 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238689 s, 43.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309703 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575387 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537261 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119287 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191288 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18835 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26375 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27861 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27617 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25836 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269667 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106417 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320316 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302751 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293391 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293083 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316071 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134705 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362458 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494437 s, 212 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687081 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584765 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671263 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627589 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348758 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753029 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549432 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267386 s, 39.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238688 s, 43.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245752 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407028 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350473 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344845 s, 30.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (17s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:07:44 (1788876464) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.39859 s, 42.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379157 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844907 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701711 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936205 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151407 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252094 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180361 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197241 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272052 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207898 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266061 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196798 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218698 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232763 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227366 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29536 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393603 s, 26.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450778 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283752 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036668 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353061 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360212 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383528 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282384 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566357 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449209 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490418 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590048 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522421 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456054 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537097 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877869 s, 11.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.372708 s, 45.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627662 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675934 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610979 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126142 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137939 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102149 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179726 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156935 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172274 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190677 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161567 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211159 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180818 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203934 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327664 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910121 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970659 s, 10.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=9 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116252 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908295 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827443 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100673 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107735 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839038 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990201 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594868 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959762 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901735 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955818 s, 11.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934214 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376592 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125483 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114003 s, 9.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.405343 s, 41.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661109 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749485 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106759 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118249 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119995 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144904 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143302 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165113 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192977 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202572 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181633 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1771 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194992 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182467 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168586 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252275 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0084337 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911837 s, 115 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150969 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048269 s, 217 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498828 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201547 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353305 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467324 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00292271 s, 359 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249193 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815864 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240485 s, 436 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585034 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214387 s, 489 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556516 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00518737 s, 202 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33499 s, 50.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365404 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891599 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897834 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127986 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133159 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150486 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544148 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182702 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127494 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226016 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237024 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170969 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248901 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204086 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19734 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191918 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425977 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566463 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428305 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579984 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508931 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134966 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129682 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750008 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373332 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672619 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060987 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634937 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00215262 s, 487 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114905 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332682 s, 315 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf405439800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588485 s, 178 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (18s) == sanityn test 77g: Change TBF type directly ============ 10:08:02 (1788876482) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (9s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:08:11 (1788876491) oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 PASS 77h (8s) == sanityn test 77i: Change rank of TBF rule ============= 10:08:19 (1788876499) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (15s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:08:34 (1788876514) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411346 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657235 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847228 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102398 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141299 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612662 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14945 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156032 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834481 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194042 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170949 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177541 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154462 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163234 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189609 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226706 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830211 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330168 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433965 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288928 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354615 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041667 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422995 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392244 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331802 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555691 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441153 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558272 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497824 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548649 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536816 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523884 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674981 s, 15.5 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.73105 s, 18.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5201 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (58s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:09:32 (1788876572) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25625 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703261 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938873 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962529 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125367 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162421 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942309 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155145 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159228 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195343 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214815 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200468 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224911 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21967 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26113 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293342 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293642 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044662 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546758 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063896 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485458 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553764 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669016 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04264 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627407 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555409 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528945 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718563 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641972 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852085 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863013 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679503 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748554 s, 14.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.879 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9773 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27097 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656015 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941703 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104306 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101652 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755032 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129557 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623982 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130712 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165935 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140717 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166956 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179396 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177941 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17049 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177486 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219722 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013881 s, 75.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270233 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284822 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035823 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423091 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358182 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045364 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039129 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397579 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041336 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391012 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474376 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500309 s, 21.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448876 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405298 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490509 s, 21.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8457 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.937 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (147s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:11:59 (1788876719) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28618 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873939 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105902 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093105 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116468 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155011 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150054 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100173 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434648 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136904 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819369 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189013 s, 5.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19786 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183359 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235666 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213149 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129123 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434391 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049657 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486092 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492206 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470367 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572274 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577504 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439356 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392873 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494352 s, 21.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410441 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626274 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521555 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504847 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584694 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636469 s, 16.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8486 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9753 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25441 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862461 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958381 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110297 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794846 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137426 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670649 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596173 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136248 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154136 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155649 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156397 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177778 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150401 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16255 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184145 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1793 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374914 s, 28.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449145 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360673 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381582 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232913 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383852 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342973 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603136 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366698 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378323 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482431 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453133 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509052 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677163 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535975 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547836 s, 19.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8601 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0059 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (149s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:14:28 (1788876868) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.374531 s, 44.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660735 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609742 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822523 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173399 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149612 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20986 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187863 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205188 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181151 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190085 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164373 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184735 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178664 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274865 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218606 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247456 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310637 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317535 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033897 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348655 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533073 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356292 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378721 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398073 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276598 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331937 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350066 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428556 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362159 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592385 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411657 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532872 s, 19.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.03397 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.892 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371613 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852927 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789493 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159203 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16926 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159512 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185521 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229418 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203206 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185685 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235674 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180461 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163033 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17722 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242771 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233287 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220861 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228286 s, 459 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627404 s, 167 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054877 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609091 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612703 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626937 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576971 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659608 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568522 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948566 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295005 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656758 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650451 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666334 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254762 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03547 s, 29.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.58763 s, 18.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87645 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.423475 s, 39.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394881 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560505 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443839 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149048 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184249 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145531 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142487 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17683 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196942 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168035 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200659 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185317 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211277 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238474 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244046 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224747 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536658 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597225 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456539 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601235 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461317 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593873 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443947 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519344 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455371 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690112 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614965 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543778 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753816 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644201 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592333 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619003 s, 16.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.91842 s, 17.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0436 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.56474 s, 29.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050141 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658416 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06754 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100799 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157524 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154237 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202685 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178061 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213927 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171357 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190873 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180048 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24981 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252916 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229483 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197485 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203846 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595444 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232376 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352715 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115451 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651312 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00851757 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106562 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441998 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061733 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565636 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582643 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626363 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994735 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063618 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664745 s, 15.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94447 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87356 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.67142 s, 18.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84299 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.883795 s, 19.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463257 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639416 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732901 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569596 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116134 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11873 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125673 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932678 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145139 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202363 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208323 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218829 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217974 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193903 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203975 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231162 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302125 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702864 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197996 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393817 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063057 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266276 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00317316 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00233537 s, 449 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651652 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718371 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391132 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677312 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569531 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847954 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0021757 s, 482 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134985 s, 77.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6929 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.564 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.418754 s, 40.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663628 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584694 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839661 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987169 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101364 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889418 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578121 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660484 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789777 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584067 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187139 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138017 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148331 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166806 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166364 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187312 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103771 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021606 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419033 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317522 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699609 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226007 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392396 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291736 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294268 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033864 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383873 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100486 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055042 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546903 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182876 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125378 s, 83.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.56851 s, 18.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8593 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.542392 s, 30.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655607 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276275 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917282 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107097 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121704 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154023 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151395 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179577 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778175 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165273 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14144 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200394 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196054 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130914 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184621 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280075 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261674 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190722 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192615 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245891 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441407 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039986 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275963 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340038 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279568 s, 37.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731305 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457969 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574915 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576773 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455225 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660935 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836118 s, 12.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0258 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86195 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93609 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85156 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (406s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 10:21:14 (1788877274) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.393603 s, 42.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023148 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588247 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17131 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17215 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194858 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192991 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179251 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171126 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184519 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209064 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181693 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22222 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198551 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22201 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217477 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188961 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360699 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509188 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270142 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04598 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370243 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358352 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452781 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472689 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367868 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544864 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045546 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456128 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624232 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505917 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552182 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583387 s, 18.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27898 s, 19.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84899 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (45s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 10:21:59 (1788877319) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.55@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.55@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.382858 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495381 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614157 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121698 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122573 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169741 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172688 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154317 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173923 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1388 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146245 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188949 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148073 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156114 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160796 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215384 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211251 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338514 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256963 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503365 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387155 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588555 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042392 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028843 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330083 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549862 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559943 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430478 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417746 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492801 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522492 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488464 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562074 s, 18.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46911 s, 19.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83582 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (42s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 10:22:41 (1788877361) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.55@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.330287 s, 50.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353095 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367781 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10926 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810176 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121756 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104333 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119655 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10859 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12543 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135915 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125848 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125742 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124468 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113872 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125241 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133488 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00853218 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249573 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106328 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338007 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375283 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135481 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171449 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172489 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182289 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216006 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239161 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205963 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354499 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335922 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275178 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252811 s, 41.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31548 s, 19.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96574 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (35s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 10:23:16 (1788877396) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.509903 s, 32.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521385 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685439 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341749 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588587 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690641 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054927 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914684 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127749 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12355 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152153 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148524 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151868 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243738 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230341 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2284 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251791 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298747 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276138 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292206 s, 35.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238022 s, 44.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239533 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217642 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288526 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308546 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324029 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342533 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321575 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365853 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323704 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397666 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385171 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483368 s, 21.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97575 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84155 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.72916 s, 18.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82933 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (82s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 10:24:38 (1788877478) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.922591 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383893 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644469 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547122 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577262 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587259 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891096 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938459 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102173 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102894 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104944 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133314 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147019 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167383 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158711 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176124 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179793 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504585 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380662 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133376 s, 78.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529286 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446186 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049308 s, 21.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490596 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251124 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260486 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026398 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540579 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678427 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458114 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319183 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00897737 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366101 s, 28.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6129 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.9575 s, 4.8 MB/s Read runtime is 23 s, speed is 4.347826 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (84s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 10:26:02 (1788877562) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.44845 s, 11.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148684 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230725 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1257 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237491 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218886 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284501 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259457 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291117 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302151 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180126 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400137 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227199 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346952 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420023 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf405439800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156579 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518543 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256147 s, 409 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110561 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687792 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636211 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00221581 s, 473 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329261 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310731 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219155 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377891 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883744 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204049 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457985 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995024 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701517 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605358 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124783 s, 8.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.5188 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0481 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.847177 s, 19.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761927 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168738 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299535 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108912 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397363 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30327 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29993 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311504 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443785 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385985 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524233 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482342 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462601 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372836 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398235 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493574 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158465 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744216 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600846 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573875 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022403 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168204 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084732 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125868 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995493 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996106 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155309 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217556 s, 48.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980043 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143851 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149876 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981998 s, 10.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9627 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95756 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9188 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97456 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (163s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 10:28:45 (1788877725) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg255-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg255-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg255-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69813/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry2 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.55,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg255-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Starting client oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf403e6c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf40bd14800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf403e6c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf40bd14800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402783 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695421 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474345 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835246 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106029 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310464 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153264 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159191 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250889 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396283 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265495 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388366 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335108 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306178 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308589 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295958 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340142 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082256 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712727 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785762 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761021 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942132 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101659 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105446 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963527 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107195 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091067 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113844 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112259 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11561 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105221 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11547 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124153 s, 8.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.07543 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19195 s, 20.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.551164 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897851 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942937 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749156 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081174 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200645 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276773 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284992 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273721 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362256 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317569 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361687 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353278 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368615 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40542 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4069 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385334 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf403e6c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9cf403e6c000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387988 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12119 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039593 s, 26.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478397 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477959 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515843 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610926 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111035 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799435 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932369 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145763 s, 7.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116329 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128531 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132491 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142227 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395294 s, 26.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.54616 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93095 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (214s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 10:32:19 (1788877939) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 10:32:38 (1788877958) On MGS 192.168.202.155, active = nodemap.active=1 On MGS 192.168.202.155, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.677006 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615461 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522394 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765928 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898258 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667662 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110773 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152417 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182908 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229349 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226595 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259544 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179288 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192874 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231979 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194684 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199645 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324077 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310573 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394853 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419727 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349973 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059733 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934376 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467617 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570986 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516664 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744975 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119545 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564733 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121605 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659927 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662027 s, 15.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.56169 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2846 s, 19.8 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.155, TBF.id = On MGS 192.168.202.155, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519083 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476739 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07311 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668397 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123634 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951955 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129844 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109368 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166386 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155691 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135569 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157074 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192776 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258457 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf403e6c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf403e6c000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206389 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259202 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224473 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045559 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473461 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368851 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606682 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497381 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340876 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672423 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435101 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428135 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425319 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352988 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411751 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402274 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444937 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148791 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626561 s, 16.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.36475 s, 14.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.60856 s, 40.2 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg255-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 On MGS 192.168.202.155, active = nodemap.active=1 On MGS 192.168.202.155, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.649202 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516751 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856086 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745087 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831592 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381279 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930311 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181334 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211586 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141697 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247266 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21107 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266833 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195647 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261782 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136678 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232808 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf403e6c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf403e6c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865372 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436122 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833977 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785783 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126859 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083464 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104319 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594469 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114662 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103477 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930411 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073564 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116285 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943534 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084783 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975786 s, 10.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg255-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg255-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.8198 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94152 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.155, TBF.id = On MGS 192.168.202.155, active = nodemap.active=0 PASS 77kj (185s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 10:35:43 (1788878143) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (8s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 10:35:51 (1788878151) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg255-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.60181 s, 291 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg255-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0453 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (53s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 10:36:44 (1788878204) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175088 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677497 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611477 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662967 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078241 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102126 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116531 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995702 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127589 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123183 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865314 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136446 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128028 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133013 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10735 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14385 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175485 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446049 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511133 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536419 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040488 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522959 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477057 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502842 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451202 s, 23.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452743 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054578 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04432 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479868 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576429 s, 18.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf403e6c000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629786 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578114 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604246 s, 17.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09899 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01824 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315753 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386151 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536193 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565606 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169394 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124597 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156293 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180596 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181504 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189677 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16432 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201757 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182262 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18776 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199046 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166961 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177282 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391847 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178838 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371482 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225892 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022711 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260656 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264149 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027147 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271274 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528364 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305352 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329146 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299424 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033297 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389264 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342913 s, 30.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg255-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg255-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20944 s, 20.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98041 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (75s) == sanityn test 77o: Changing rank should not panic ====== 10:37:59 (1788878279) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (6s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 10:38:05 (1788878285) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (49s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 10:38:54 (1788878334) mds.MDS.mdt.nrs_policies=tbf oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg255-client: oleg255-server: ssh exited with exit code 36 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 oleg255-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg255-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg255-client: oleg255-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 10:39:08 (1788878348) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 275763 kill md_thread xargs PASS 77r (44s) == sanityn test 77s: Check TBF LRU shrinker ============== 10:39:52 (1788878392) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (23s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 10:40:15 (1788878415) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (5s) == sanityn test 79: xattr: intent error ================== 10:40:20 (1788878420) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936047 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:40:21.000000000 -0400 Modify: 2026-09-08 10:40:21.000000000 -0400 Change: 2026-09-08 10:40:21.000000000 -0400 Birth: 2026-09-08 10:40:21.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 10:40:33 (1788878433) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 10:40:35 (1788878435) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 10:40:36 (1788878436) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 10:40:37 (1788878437) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 10:40:38 (1788878438) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 10:40:39 (1788878439) total: 2500 open/close in 6.95 seconds: 359.78 ops/second snapshot_time 1788878472.247328947 secs.nsecs start_time 1788878441.362603649 secs.nsecs elapsed_time 30.884725298 secs.nsecs open 2505 samples [usecs] 103 4457 1243776 713688852 close 2504 samples [usecs] 64 1803 322507 53189871 mknod 2501 samples [usecs] 221 4363 1113844 579729754 unlink 2500 samples [usecs] 321 13182 1831533 2208618721 mkdir 2 samples [usecs] 849 1213 2062 2192170 rmdir 2 samples [usecs] 1993 3716 5709 17780705 rename 2500 samples [usecs] 304 13008 4981762 17313562448 getattr 3761 samples [usecs] 6 1463 107914 9529458 parallel_rename_file 2500 samples [usecs] 304 13008 4981762 17313562448 crossdir_rename 2500 samples [usecs] 304 13008 4981762 17313562448 readpage 14 samples [usecs] 1859 9944 67062 419853364 PASS 81d (34s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 10:41:13 (1788878473) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7647 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7647 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 10:41:16 (1788878476) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 10:41:17 (1788878477) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 10:41:24 (1788878484) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788878485 Finishing test test1 at 1788878486 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 10:41:27 (1788878487) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 10:41:29 (1788878489) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 10:41:30 (1788878490) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 10:41:31 (1788878491) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6159 0x180f 0x240000400 1 6118 0x17e6 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6119 0x17e7 0x280000400 0 6160 0x1810 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 10:41:40 (1788878500) osc.lustre-OST0000-osc-ffff9cf403e6c000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9cf40bd14800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cf403e6c000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cf40bd14800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0440919 s, 186 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09316 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 295235 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0136915 s, 299 kB/s osc.lustre-OST0000-osc-ffff9cf403e6c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf40bd14800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf403e6c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf40bd14800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 10:41:50 (1788878510) llite.lustre-ffff9cf403e6c000.fast_read=0 llite.lustre-ffff9cf40bd14800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.017823 s, 118 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9cf403e6c000.fast_read=1 llite.lustre-ffff9cf40bd14800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 10:42:05 (1788878525) llite.lustre-ffff9cf403e6c000.fast_read=0 llite.lustre-ffff9cf40bd14800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0018668 s, 6.6 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9cf403e6c000.fast_read=1 llite.lustre-ffff9cf40bd14800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 10:42:13 (1788878533) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 10:42:14 (1788878534) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00154421 s, 2.7 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 10:42:16 (1788878536) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.03706 s, 56.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 10:42:19 (1788878539) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0434865 s, 48.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 10:42:21 (1788878541) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00180518 s, 6.6 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 10:42:23 (1788878543) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00136583 s, 3.0 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 10:42:26 (1788878546) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00185222 s, 2.2 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 10:42:28 (1788878548) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00214451 s, 1.9 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 10:42:31 (1788878551) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 d0 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x10d0:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (2s) == sanityn test 103: Test size correctness with lockahead ========================================================== 10:42:33 (1788878553) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788878554 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788878557 Finishing test test23 at 1788878557 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 10:42:41 (1788878561) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 10:42:42 (1788878562) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 10:43:04 (1788878584) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 10:43:06 (1788878586) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00545507 s, 188 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938583 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:43:06.000000000 -0400 Modify: 2026-09-08 10:43:06.000000000 -0400 Change: 2026-09-08 10:43:06.000000000 -0400 Birth: 2026-09-08 10:43:06.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938583 -rw-r--r-- 1 0 0 1788878586 1788878586 1788878586 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938583 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:43:06.000000000 -0400 Modify: 2026-09-08 10:43:06.000000000 -0400 Change: 2026-09-08 10:43:06.000000000 -0400 Birth: 2026-09-08 10:43:06.000000000 -0400 1788878586 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 10:43:08 (1788878588) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 10:43:10 (1788878590) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0840873 s, 125 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6126 0x17ee 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 10:43:14 (1788878594) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.106007 s, 98.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6169 0x1819 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7647 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 10:43:23 (1788878603) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00187965 s, 5.3 MB/s llite.lustre-ffff9cf403e6c000.hybrid_io=0 llite.lustre-ffff9cf40bd14800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.21381 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 10:43:29 (1788878609) 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 2 Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 192.168.202.155@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: kmemleak disabled oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre2 PASS 109 (35s) == sanityn test 110: do not grant another lock on resend ========================================================== 10:44:04 (1788878644) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 10:44:05.000000000 -0400 Modify: 2026-09-08 10:44:05.000000000 -0400 Change: 2026-09-08 10:44:05.000000000 -0400 Birth: 2026-09-08 10:44:05.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg255-client.virtnet oleg255-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 10:46:50 (1788878810) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 10:46:51 (1788878811) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 10:46:52 (1788878812) lustre-OST0000-osc-ffff9cf404bfa800 active. lustre-OST0001-osc-ffff9cf404bfa800 active. lustre-OST0000-osc-ffff9cf404bfa800 active. lustre-OST0001-osc-ffff9cf404bfa800 active. MGC192.168.202.155@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 10:46:54 (1788878814) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 10:46:55 (1788878815) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 10:46:57 (1788878817) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 10:46:58 (1788878818) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00629313 s, 3.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0240506 s, 852 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 122: directory size is consistent across mounts ========================================================== 10:47:02 (1788878822) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 10:47:05 (1788878825) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0899 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0815 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0869 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0844 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0866 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0895 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0956 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1374 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1293 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1399 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1446 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1454 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1584 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1686 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2227 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2256 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2156 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2261 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf40343b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf40343b800.ost_server_uuid in IDLE state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 9029 sec ============== 10:48:13 (1788878893) === sanityn: start cleanup 10:48:14 (1788878894) === Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg255-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 10:49:01 (1788878941) ===