-----============= acceptance-small: sanityn ============----- Thu Jul 30 02:52:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg452-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 02:53:21 (1785394401) === Starting client oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Mount client oleg452-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Started clients oleg452-client.virtnet: 192.168.204.152@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg452-client.virtnet: executing check_config_client /mnt/lustre oleg452-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg452-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9316c5236800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9316c79f6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316c5236800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316c79f6000.idle_timeout=debug disable quota as required oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 02:54:15 (1785394455) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793731 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:54:18 (1785394458) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:54:28 (1785394468) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (12s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:54:40 (1785394480) -rw-r--r-- 1 root root 0 Jul 30 02:54 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:54:49 (1785394489) -rw-r--r-- 1 root root 0 Jul 30 02:54 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:54:57 (1785394497) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (9s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:55:06 (1785394506) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (15s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:55:21 (1785394521) total 1 -rwxrwxrwx 1 root root 0 Jul 30 02:54 f2a -rwxrwxrwx 1 root root 0 Jul 30 02:54 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 30 02:54 f2a -rwxrwxrwx 1 root root 0 Jul 30 02:54 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0737385 s, 6.9 kB/s PASS 2e (14s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:55:35 (1785394535) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 02:55:40 (1785394540) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.603609 s, 3.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:55:50 (1785394550) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:55:58 (1785394558) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:56:07 (1785394567) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 02:56:16 (1785394576) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:56:26 (1785394586) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (9s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:56:35 (1785394595) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (9s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:56:44 (1785394604) PASS 9a (11s) == sanityn test 9b: append to striped sparse file ======== 02:56:55 (1785394615) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468184 s, 22.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0908065 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:57:03 (1785394623) 1+0 records in 1+0 records out 1 byte copied, 0.0129533 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0295039 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0156095 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127606 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00994628 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00857589 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010693 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00735015 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00973807 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00815851 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00901583 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00920463 s, 0.1 kB/s PASS 10a (11s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:57:14 (1785394634) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0219814 s, 140 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.057841 s, 70.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000544862 s, 5.6 MB/s PASS 10b (10s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:57:24 (1785394644) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (13s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:57:37 (1785394657) warning: '-runtime' deprecated, use '-t runtime' instead - link 207 (time 1785394671.79 total 10.04 last 20.61) - link 422 (time 1785394681.81 total 20.06 last 21.45) - link 639 (time 1785394691.85 total 30.10 last 21.61) - link 848 (time 1785394701.88 total 40.13 last 20.84) - link 1082 (time 1785394711.95 total 50.20 last 23.24) - link 1286 (time 1785394721.99 total 60.25 last 20.31) - link 1510 (time 1785394732.02 total 70.28 last 22.33) - link 1768 (time 1785394742.06 total 80.31 last 25.71) - link 1992 (time 1785394752.11 total 90.36 last 22.29) total: 2198 link in 100.05 seconds: 21.97 ops/second using seed 1723359367 running for 100 seconds - stat 2577 (time 1785394673 ; total 11 ; last 11) - stat 6405 (time 1785394684 ; total 22 ; last 11) - stat 9337 (time 1785394695 ; total 33 ; last 11) - stat 10000 (time 1785394697 ; total 35 ; last 2) - stat 13214 (time 1785394708 ; total 46 ; last 11) - stat 15459 (time 1785394719 ; total 57 ; last 11) - stat 17490 (time 1785394730 ; total 68 ; last 11) - stat 19493 (time 1785394741 ; total 79 ; last 11) - stat 20000 (time 1785394744 ; total 82 ; last 3) - stat 21800 (time 1785394755 ; total 93 ; last 11) total: 22949 stats in 100 seconds: 229.490005 stats/second - unlinked 0 (time 1785394712 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2198) error: No such file or directory total: 2197 unlinks in 89 seconds: 24.685392 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21439) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21441) - No such process PASS 12 (152s) == sanityn test 13: test directory page revocation ======= 03:00:09 (1785394809) f13 f13 PASS 13 (13s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 03:00:22 (1785394822) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 22630 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (9s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 03:00:31 (1785394831) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 23210 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 03:00:41 (1785394841) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23789 Terminated $DIR1/$tdir/sleep 60 PASS 14b (10s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 03:00:50 (1785394850) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24372 Terminated $DIR1/$tdir/sleep 60 PASS 14c (10s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 03:01:00 (1785394860) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24968 Terminated $DIR1/$tdir/sleep 60 PASS 14d (13s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 03:01:13 (1785394873) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (59s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 03:02:12 (1785394932) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.12658 s, 7.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.06558 s, 7.9 MB/s Chance of close/open is 1 in 50 Seed set to 4940 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1785394942.225327 write 0x6a45fe thru 0x6ad2ef (0x8cf2 bytes) 000003[0] 1785394942.244730 mapread 0x50c26c thru 0x50fce3 (0x3a78 bytes) 000004[0] 1785394942.280404 mapwrite 0x30a345 thru 0x312c47 (0x8903 bytes) 000005[0] 1785394942.618316 mapwrite 0x93d53e thru 0x942973 (0x5436 bytes) truncating to largest ever: 0x206295 000007[1] 1785394943.054331 trunc from 0x942974 to 0x206294 (0x73c6df bytes) truncating to largest ever: 0x7acfb4 000009[0] 1785394943.242212 trunc from 0x206295 to 0x7acfb3 (0x5a6d1f bytes) 000010[1] 1785394943.403746 write 0x3f8b29 thru 0x3f9353 (0x82b bytes) 000011[0] 1785394943.505461 mapread 0x17d5cc thru 0x17f762 (0x2197 bytes) 000012[0] 1785394943.710906 mapread 0x3612b0 thru 0x3634c2 (0x2213 bytes) 000013[1] 1785394943.772412 trunc from 0x7acfb4 to 0x75df0b (0x4f0a8 bytes) 000016[1] 1785394944.006622 mapread 0x3e3c69 thru 0x3edf2e (0xa2c6 bytes) 000017[1] 1785394944.073719 read 0x22587f thru 0x22e799 (0x8f1b bytes) 000018[1] 1785394944.130945 trunc from 0x75df0c to 0x11a1f2 (0x643d19 bytes) 000019[1] 1785394944.179158 read 0x0a986a thru 0x0b259f (0x8d36 bytes) 000021[0] 1785394944.293858 trunc from 0x11a1f3 to 0x1f76b5 (0xdd4c3 bytes) 000022[0] 1785394944.391241 mapread 0x0f0a4c thru 0x0f4143 (0x36f8 bytes) 000026[1] 1785394944.455566 write 0x6898bd thru 0x694768 (0xaeac bytes) 000027[0] 1785394944.489112 mapread 0x029d91 thru 0x0394c7 (0xf737 bytes) 000028[0] 1785394944.550462 write 0x04f7ce thru 0x0525f5 (0x2e28 bytes) 000030[1] 1785394944.607458 trunc from 0x694769 to 0x721785 (0x8d01d bytes) 000031[0] 1785394944.676596 trunc from 0x721786 to 0x18c889 (0x594efc bytes) 000033[0] 1785394944.821022 mapread 0x103c65 thru 0x1055b6 (0x1952 bytes) 000034[1] 1785394944.840249 mapread 0x0289d0 thru 0x03362b (0xac5c bytes) 000041[0] 1785394945.155918 write 0x175a8a thru 0x17cadf (0x7056 bytes) 000042[1] 1785394945.309568 write 0x9f766c thru 0x9fffff (0x8994 bytes) 000044[1] 1785394945.473158 trunc from 0xa00000 to 0x77a328 (0x285cd7 bytes) 000046[0] 1785394945.566891 trunc from 0x77a329 to 0x2e30dd (0x49724b bytes) 000049[1] 1785394945.718226 write 0x3d22ab thru 0x3de6ac (0xc402 bytes) 000050[0] 1785394945.745636 write 0x3464e6 thru 0x35214e (0xbc69 bytes) 000055[0] 1785394945.884228 write 0x14ac4d thru 0x157902 (0xccb6 bytes) 000056[0] 1785394945.920853 mapwrite 0x4c931a thru 0x4cb941 (0x2628 bytes) 000058[0] 1785394946.025879 mapwrite 0x3bc290 thru 0x3bceb5 (0xc26 bytes) 000059[1] 1785394946.093629 write 0x5a413d thru 0x5ad120 (0x8fe4 bytes) 000064[1] 1785394946.169839 read 0x537e4a thru 0x547c95 (0xfe4c bytes) 000065[1] 1785394946.187874 write 0x702717 thru 0x703f59 (0x1843 bytes) 000066[1] 1785394946.203914 trunc from 0x703f5a to 0x0498a7 (0x6ba6b2 bytes) 000071[0] 1785394946.272085 read 0x03aa5f thru 0x041d22 (0x72c4 bytes) 000072[1] 1785394946.297612 mapwrite 0x2128e3 thru 0x21670a (0x3e28 bytes) 000073[1] 1785394946.441327 mapwrite 0x448e23 thru 0x455b45 (0xcd23 bytes) 000076[1] 1785394946.780761 trunc from 0x455b46 to 0x2b7275 (0x19e8d0 bytes) 000077[0] 1785394946.818675 write 0x4c1ee6 thru 0x4c1ff9 (0x114 bytes) 000079[1] 1785394946.924943 write 0x714345 thru 0x7155f6 (0x12b2 bytes) 000080[0] 1785394947.008369 trunc from 0x7155f7 to 0x1d7b88 (0x53da6e bytes) 000081[1] 1785394947.148698 read 0x16cc23 thru 0x16d06f (0x44d bytes) 000084[0] 1785394947.379522 mapread 0x02200e thru 0x024f60 (0x2f53 bytes) 000086[0] 1785394947.477531 write 0x831ec4 thru 0x83331f (0x145c bytes) 000097[0] 1785394947.964143 mapread 0x4c7d34 thru 0x4c832d (0x5fa bytes) 000098[0] 1785394947.998520 mapwrite 0x74f3a2 thru 0x757fc8 (0x8c27 bytes) 000099[1] 1785394948.510809 trunc from 0x833320 to 0x6a41de (0x18f141 bytes) truncating to largest ever: 0x9c3a64 000100[1] 1785394948.844429 trunc from 0x6a41df to 0x9c3a63 (0x31f885 bytes) 000101[0] 1785394949.087457 read 0x04e711 thru 0x05db36 (0xf426 bytes) 000105[0] 1785394949.411768 write 0x5391a8 thru 0x53a60a (0x1463 bytes) 000106[0] 1785394949.485014 trunc from 0x9c3a64 to 0x0db472 (0x8e85f1 bytes) 000107[1] 1785394949.680299 trunc from 0x0db473 to 0x90789c (0x82c42a bytes) 000108[1] 1785394949.928957 mapwrite 0x9084ea thru 0x90e865 (0x637c bytes) 000114[0] 1785394950.421632 read 0x8bad14 thru 0x8c7078 (0xc365 bytes) 000116[1] 1785394950.565501 write 0x41fff7 thru 0x422de0 (0x2dea bytes) 000118[0] 1785394950.638606 mapwrite 0x46bf5d thru 0x47ac2f (0xecd3 bytes) 000119[0] 1785394951.023684 mapread 0x652ad2 thru 0x6552e2 (0x2811 bytes) truncating to largest ever: 0x9ce936 000121[1] 1785394951.154616 trunc from 0x90e866 to 0x9ce935 (0xc00d0 bytes) 000122[1] 1785394951.400493 write 0x39991f thru 0x39a77f (0xe61 bytes) 000124[0] 1785394951.467665 mapread 0x2308f1 thru 0x23ea32 (0xe142 bytes) 000125[0] 1785394951.636970 mapread 0x261e1e thru 0x2644a8 (0x268b bytes) 000127[0] 1785394951.733562 mapread 0x1eb377 thru 0x1f5ee1 (0xab6b bytes) 000131[0] 1785394951.891054 write 0x20f697 thru 0x21b5e4 (0xbf4e bytes) 000132[1] 1785394952.004736 write 0x84b076 thru 0x84cf49 (0x1ed4 bytes) 000133[1] 1785394952.095857 write 0x401921 thru 0x40663d (0x4d1d bytes) 000134[1] 1785394952.225281 write 0x7d021c thru 0x7de3de (0xe1c3 bytes) truncating to largest ever: 0x9df71f 000135[0] 1785394952.323172 trunc from 0x9ce936 to 0x9df71e (0x10de9 bytes) 000136[1] 1785394952.444304 read 0x17fd65 thru 0x186113 (0x63af bytes) 000139[0] 1785394952.508207 write 0x453a9b thru 0x457c18 (0x417e bytes) 000141[0] 1785394952.588690 mapread 0x2d9c7b thru 0x2dc850 (0x2bd6 bytes) 000144[0] 1785394952.627207 write 0x711205 thru 0x71391f (0x271b bytes) 000146[1] 1785394952.783539 mapread 0x2fa62b thru 0x2fc1a1 (0x1b77 bytes) 000147[1] 1785394952.856270 read 0x324920 thru 0x32ad4c (0x642d bytes) 000148[1] 1785394952.943189 mapread 0x577831 thru 0x57b3b6 (0x3b86 bytes) 000155[0] 1785394953.149983 read 0x8046d8 thru 0x807fd1 (0x38fa bytes) 000159[0] 1785394953.192382 trunc from 0x9df71f to 0x420142 (0x5bf5dc bytes) 000161[0] 1785394953.387505 read 0x1e216f thru 0x1e36b3 (0x1545 bytes) 000164[0] 1785394953.504084 mapread 0x09fab5 thru 0x0aaf6e (0xb4ba bytes) 000168[0] 1785394953.784750 mapwrite 0x47d9c1 thru 0x48a17d (0xc7bd bytes) 000170[1] 1785394954.442854 mapread 0x2ff357 thru 0x3075ef (0x8299 bytes) 000171[0] 1785394954.713206 read 0x3c70e8 thru 0x3c9191 (0x20aa bytes) 000179[0] 1785394954.976530 write 0x549a4d thru 0x54bf39 (0x24ed bytes) 000181[0] 1785394955.062675 mapread 0x0b0e97 thru 0x0b53b0 (0x451a bytes) truncating to largest ever: 0x9e58cb 000182[0] 1785394955.168113 trunc from 0x54bf3a to 0x9e58ca (0x499991 bytes) 000184[0] 1785394955.317676 trunc from 0x9e58cb to 0x278d9c (0x76cb2e bytes) 000185[1] 1785394955.453901 write 0x0e6bd6 thru 0x0eb747 (0x4b72 bytes) 000186[1] 1785394955.554361 write 0x0b60f3 thru 0x0b92f3 (0x3201 bytes) 000188[0] 1785394955.674887 write 0x2ed9fa thru 0x2f5a7a (0x8081 bytes) 000190[1] 1785394955.833766 read 0x249d6e thru 0x24f444 (0x56d7 bytes) 000195[0] 1785394956.266959 mapwrite 0x763556 thru 0x769514 (0x5fbf bytes) 000197[1] 1785394956.498160 mapread 0x3fb1ea thru 0x3fffcb (0x4de2 bytes) 000198[1] 1785394956.615409 read 0x52b5e2 thru 0x5306f6 (0x5115 bytes) 000202[1] 1785394956.641046 mapwrite 0x532b6b thru 0x537e08 (0x529e bytes) 000204[0] 1785394956.752667 read 0x43f6a4 thru 0x4421a5 (0x2b02 bytes) 000205[1] 1785394956.774213 trunc from 0x769515 to 0x20156f (0x567fa5 bytes) 000206[1] 1785394956.827530 read 0x1cd451 thru 0x1d5823 (0x83d3 bytes) 000207[1] 1785394956.906807 read 0x0a5036 thru 0x0ad1f9 (0x81c4 bytes) 000208[1] 1785394956.907711 read 0x0a2a7b thru 0x0b0fe3 (0xe569 bytes) 000210[0] 1785394956.933332 mapread 0x1c1228 thru 0x1d0e21 (0xfbfa bytes) 000211[0] 1785394957.155138 mapwrite 0x58e05f thru 0x58e7a5 (0x747 bytes) 000212[1] 1785394957.592988 read 0x47a050 thru 0x48143a (0x73eb bytes) 000213[0] 1785394957.760499 write 0x193334 thru 0x19d58b (0xa258 bytes) 000214[1] 1785394957.820205 mapwrite 0x1234f9 thru 0x13270b (0xf213 bytes) 000215[1] 1785394957.934423 trunc from 0x58e7a6 to 0x3ad686 (0x1e111f bytes) 000216[0] 1785394957.960860 mapread 0x189271 thru 0x18bd6c (0x2afc bytes) 000217[0] 1785394958.003405 read 0x1194f5 thru 0x11fcdb (0x67e7 bytes) 000218[1] 1785394958.093351 read 0x248015 thru 0x24844c (0x438 bytes) 000220[1] 1785394958.188895 mapwrite 0x03db71 thru 0x049fc6 (0xc456 bytes) 000221[0] 1785394958.775434 write 0x141524 thru 0x146a7b (0x5558 bytes) 000222[0] 1785394958.954340 read 0x131a17 thru 0x13d5ad (0xbb97 bytes) 000223[0] 1785394959.054595 mapread 0x13a4c4 thru 0x13c3fc (0x1f39 bytes) 000224[0] 1785394959.077559 mapread 0x14e662 thru 0x15ac12 (0xc5b1 bytes) 000225[1] 1785394959.096643 write 0x32ef7c thru 0x338055 (0x90da bytes) 000226[0] 1785394959.132320 mapwrite 0x5c01bd thru 0x5c465f (0x44a3 bytes) 000227[0] 1785394959.465115 write 0x7ad35b thru 0x7b0fdd (0x3c83 bytes) 000228[1] 1785394959.510468 write 0x99c8df thru 0x99d88c (0xfae bytes) 000232[1] 1785394959.687648 write 0x7456c5 thru 0x755357 (0xfc93 bytes) 000237[0] 1785394960.266326 trunc from 0x99d88d to 0x633848 (0x36a044 bytes) 000238[1] 1785394960.507497 mapwrite 0x59be9b thru 0x59fe7e (0x3fe4 bytes) 000239[0] 1785394960.839150 write 0x2d7873 thru 0x2df96a (0x80f8 bytes) 000241[1] 1785394960.895273 mapwrite 0x5e96d2 thru 0x5f69c8 (0xd2f7 bytes) 000242[0] 1785394961.011459 trunc from 0x633849 to 0x397f75 (0x29b8d3 bytes) 000245[0] 1785394961.095464 mapwrite 0x5befc2 thru 0x5bfc6c (0xcab bytes) 000247[0] 1785394961.210466 trunc from 0x5bfc6d to 0x638ba2 (0x78f36 bytes) 000249[0] 1785394961.256670 trunc from 0x638ba3 to 0x4b4cce (0x183ed4 bytes) 000252[1] 1785394961.320209 write 0x74ab84 thru 0x74f240 (0x46bd bytes) 000254[1] 1785394961.400541 mapwrite 0x0af1b5 thru 0x0b8712 (0x955e bytes) 000255[0] 1785394961.805153 mapread 0x165e0e thru 0x16bf38 (0x612b bytes) 000259[1] 1785394962.052681 mapwrite 0x68513e thru 0x690da2 (0xbc65 bytes) 000261[1] 1785394962.421632 write 0x6e985b thru 0x6f74e8 (0xdc8e bytes) 000265[0] 1785394962.670066 mapwrite 0x19c636 thru 0x1a8749 (0xc114 bytes) 000267[1] 1785394963.190152 mapread 0x65e9c6 thru 0x66a2b1 (0xb8ec bytes) 000268[0] 1785394963.463129 trunc from 0x74f241 to 0x4234e6 (0x32bd5a bytes) 000269[0] 1785394963.640700 mapread 0x07cce1 thru 0x089dee (0xd10e bytes) 000272[0] 1785394963.973188 read 0x04a991 thru 0x0540a9 (0x9719 bytes) 000274[1] 1785394964.097473 read 0x1990d4 thru 0x19a3be (0x12eb bytes) 000277[0] 1785394964.267475 mapwrite 0x55e81f thru 0x560d99 (0x257b bytes) 000279[0] 1785394964.692242 write 0x96daf4 thru 0x97248e (0x499b bytes) 000282[0] 1785394964.801267 write 0x3ce24b thru 0x3d263f (0x43f5 bytes) 000292[0] 1785394965.113930 mapwrite 0x4424f2 thru 0x446f9a (0x4aa9 bytes) 000293[0] 1785394965.363387 write 0x85fe03 thru 0x866e2c (0x702a bytes) 000294[0] 1785394965.426190 mapread 0x039c16 thru 0x047c92 (0xe07d bytes) 000298[1] 1785394965.597617 mapwrite 0x36251d thru 0x371131 (0xec15 bytes) 000302[1] 1785394965.874395 trunc from 0x97248f to 0x020184 (0x95230a bytes) 000303[0] 1785394965.909303 read 0x00a727 thru 0x019d68 (0xf642 bytes) 000305[1] 1785394965.956195 mapread 0x0031a2 thru 0x010a4d (0xd8ac bytes) 000307[1] 1785394966.086864 mapread 0x01b216 thru 0x020184 (0x4f6f bytes) 000308[1] 1785394966.103457 write 0x0ba198 thru 0x0c9a40 (0xf8a9 bytes) 000311[0] 1785394966.177061 write 0x4d5696 thru 0x4d749b (0x1e06 bytes) 000312[0] 1785394966.228934 write 0x2a3316 thru 0x2b08f0 (0xd5db bytes) 000315[1] 1785394966.268260 read 0x0bc076 thru 0x0c20d6 (0x6061 bytes) 000316[1] 1785394966.378669 write 0x9e7aa4 thru 0x9f7a7b (0xffd8 bytes) 000317[0] 1785394966.432251 trunc from 0x9f7a7c to 0x177e2b (0x87fc50 bytes) 000320[1] 1785394966.533200 write 0x1a91e6 thru 0x1b221f (0x903a bytes) 000325[1] 1785394966.679527 read 0x04e4a2 thru 0x05a7c3 (0xc322 bytes) 000326[0] 1785394966.774301 mapwrite 0x2d9e55 thru 0x2e21d6 (0x8382 bytes) 000327[1] 1785394967.066931 read 0x122223 thru 0x131a47 (0xf825 bytes) 000328[0] 1785394967.163464 read 0x17b382 thru 0x17d0ef (0x1d6e bytes) 000329[0] 1785394967.228222 trunc from 0x2e21d7 to 0x299fa3 (0x48233 bytes) 000331[1] 1785394967.279909 mapread 0x09b97d thru 0x0a81bc (0xc840 bytes) 000335[0] 1785394967.409497 mapread 0x20f805 thru 0x21684b (0x7047 bytes) 000336[1] 1785394967.489299 mapwrite 0x2c459c thru 0x2d20d5 (0xdb3a bytes) 000337[1] 1785394967.785360 mapwrite 0x9d1fd6 thru 0x9e19a4 (0xf9cf bytes) 000339[1] 1785394968.033921 read 0x803d81 thru 0x8115b4 (0xd834 bytes) 000340[0] 1785394968.096045 mapread 0x9ceaa1 thru 0x9d4ee0 (0x6440 bytes) 000342[0] 1785394968.300739 read 0x6ca225 thru 0x6d93cc (0xf1a8 bytes) 000344[1] 1785394968.377396 trunc from 0x9e19a5 to 0x2b8311 (0x729693 bytes) 000346[1] 1785394968.614467 read 0x2b690e thru 0x2b8311 (0x1a04 bytes) 000347[1] 1785394968.797662 mapread 0x19043e thru 0x1a0079 (0xfc3c bytes) 000353[1] 1785394968.860762 write 0x768165 thru 0x76fa46 (0x78e2 bytes) 000354[1] 1785394968.875807 write 0x3acdc1 thru 0x3b4566 (0x77a6 bytes) 000355[1] 1785394968.950201 write 0x1ded65 thru 0x1eb6e3 (0xc97f bytes) 000357[1] 1785394969.049352 write 0x6549b2 thru 0x658a09 (0x4058 bytes) 000362[0] 1785394969.182195 mapwrite 0x7bd1b2 thru 0x7c5078 (0x7ec7 bytes) 000363[0] 1785394969.494668 read 0x04e29e thru 0x057344 (0x90a7 bytes) 000364[0] 1785394969.597425 write 0x520dda thru 0x52a4bc (0x96e3 bytes) 000365[1] 1785394969.698939 mapread 0x19f8e6 thru 0x1aae90 (0xb5ab bytes) 000367[1] 1785394969.938386 mapwrite 0x94324d thru 0x94d822 (0xa5d6 bytes) 000370[1] 1785394970.487492 mapwrite 0x7ca786 thru 0x7d20bf (0x793a bytes) 000373[0] 1785394970.649077 mapread 0x03a994 thru 0x049d30 (0xf39d bytes) 000375[1] 1785394970.751120 mapread 0x8c31e3 thru 0x8c4834 (0x1652 bytes) 000376[0] 1785394970.771256 trunc from 0x94d823 to 0x878b6e (0xd4cb4 bytes) 000377[0] 1785394970.857287 write 0x1dfd77 thru 0x1ebc13 (0xbe9d bytes) 000378[1] 1785394970.876674 mapread 0x6f8ec4 thru 0x708b3c (0xfc79 bytes) 000380[1] 1785394971.256551 read 0x5975ec thru 0x59a06d (0x2a82 bytes) 000381[0] 1785394971.270099 trunc from 0x878b6f to 0x69264e (0x1e6520 bytes) 000383[1] 1785394971.404639 mapread 0x043446 thru 0x04ee56 (0xba11 bytes) 000385[0] 1785394971.580679 mapread 0x3362c7 thru 0x34034c (0xa086 bytes) 000388[0] 1785394971.697141 write 0x216cc5 thru 0x220e2d (0xa169 bytes) 000389[0] 1785394971.829823 write 0x79b2a1 thru 0x79eb69 (0x38c9 bytes) 000393[0] 1785394971.871067 write 0x5a721c thru 0x5ae38d (0x7172 bytes) 000394[0] 1785394971.897268 read 0x311294 thru 0x31c9f9 (0xb766 bytes) 000395[1] 1785394971.924607 trunc from 0x79eb6a to 0x707f3a (0x96c2f bytes) 000398[1] 1785394972.319023 read 0x1f7377 thru 0x1fd999 (0x6623 bytes) 000400[1] 1785394972.420501 read 0x053894 thru 0x055bc3 (0x2330 bytes) 000402[0] 1785394972.549733 write 0x521a62 thru 0x524ecb (0x346a bytes) 000404[1] 1785394972.690463 trunc from 0x707f3b to 0x01cd7e (0x6eb1bc bytes) 000407[0] 1785394972.898839 mapread 0x0061f0 thru 0x00c829 (0x663a bytes) 000409[0] 1785394973.121054 read 0x001ba6 thru 0x00e830 (0xcc8b bytes) 000410[1] 1785394973.141432 read 0x00e82e thru 0x01be93 (0xd666 bytes) 000411[0] 1785394973.253148 write 0x75d2d3 thru 0x76a75d (0xd48b bytes) 000412[1] 1785394973.383762 write 0x934af8 thru 0x940c3f (0xc148 bytes) 000413[1] 1785394973.521026 trunc from 0x940c40 to 0x7abd10 (0x194f2f bytes) 000415[0] 1785394973.845119 trunc from 0x7abd11 to 0x3fcffb (0x3aed15 bytes) 000416[1] 1785394973.927462 write 0x4461ea thru 0x452aae (0xc8c5 bytes) 000417[1] 1785394974.021207 mapwrite 0x46bf22 thru 0x46e4fb (0x25da bytes) 000418[1] 1785394974.532686 write 0x6de5a9 thru 0x6e08d0 (0x2328 bytes) 000420[0] 1785394974.609616 write 0x2189c3 thru 0x222f8a (0xa5c8 bytes) 000421[0] 1785394974.888167 mapread 0x5e5c92 thru 0x5f4d1d (0xf08c bytes) 000422[1] 1785394975.059653 mapwrite 0x31282c thru 0x31a897 (0x806c bytes) 000423[1] 1785394975.619684 mapread 0x29fcda thru 0x2a4121 (0x4448 bytes) 000425[0] 1785394975.867466 mapread 0x1f1b8e thru 0x1f896d (0x6de0 bytes) 000429[0] 1785394976.199077 write 0x8d45c7 thru 0x8d874f (0x4189 bytes) 000431[0] 1785394976.302374 write 0x561e79 thru 0x569fc9 (0x8151 bytes) 000432[0] 1785394976.434289 write 0x7247a3 thru 0x729de2 (0x5640 bytes) 000435[0] 1785394976.581772 mapread 0x0451a5 thru 0x049e79 (0x4cd5 bytes) 000436[1] 1785394976.657262 trunc from 0x8d8750 to 0x9a1a8d (0xc933e bytes) 000439[1] 1785394976.831512 read 0x30505e thru 0x30577c (0x71f bytes) 000441[0] 1785394976.931558 mapwrite 0x84d42e thru 0x85c1eb (0xedbe bytes) 000442[0] 1785394977.246381 trunc from 0x9a1a8e to 0x164865 (0x83d228 bytes) 000443[1] 1785394977.316664 trunc from 0x164866 to 0x5ebf1d (0x4876b8 bytes) 000444[0] 1785394977.418281 read 0x533adf thru 0x539ca0 (0x61c2 bytes) 000445[1] 1785394977.450100 mapread 0x4cb9c0 thru 0x4d6585 (0xabc6 bytes) 000446[1] 1785394977.498167 mapwrite 0x9e18cc thru 0x9e283e (0xf73 bytes) 000448[1] 1785394977.643899 write 0x47cf36 thru 0x484010 (0x70db bytes) 000457[1] 1785394977.732062 mapread 0x7c8e47 thru 0x7cfe7f (0x7039 bytes) 000459[1] 1785394977.755232 mapwrite 0x1eddeb thru 0x1f0321 (0x2537 bytes) 000460[0] 1785394977.848482 write 0x2a7400 thru 0x2af81e (0x841f bytes) 000461[1] 1785394977.917665 mapread 0x1f9021 thru 0x203278 (0xa258 bytes) 000466[1] 1785394978.082399 mapread 0x97b8e2 thru 0x981f65 (0x6684 bytes) 000473[1] 1785394978.114541 trunc from 0x9e283f to 0x1d768b (0x80b1b3 bytes) 000475[0] 1785394978.147303 write 0x69eeab thru 0x6a6d63 (0x7eb9 bytes) 000478[0] 1785394978.218040 write 0x43408e thru 0x43b90c (0x787f bytes) 000480[1] 1785394978.258183 mapread 0x4bdfac thru 0x4c0618 (0x266d bytes) 000484[0] 1785394978.352135 mapwrite 0x05628a thru 0x05a1db (0x3f52 bytes) 000486[1] 1785394978.495214 mapwrite 0x4b217a thru 0x4b79cb (0x5852 bytes) 000487[1] 1785394978.583540 mapread 0x289ba4 thru 0x28bd9d (0x21fa bytes) 000488[1] 1785394978.626917 read 0x331a88 thru 0x33b548 (0x9ac1 bytes) 000490[0] 1785394978.688756 write 0x993dd6 thru 0x9964ef (0x271a bytes) 000493[1] 1785394978.743675 mapread 0x868e38 thru 0x86949d (0x666 bytes) 000495[0] 1785394978.809337 trunc from 0x9964f0 to 0x0396b5 (0x95ce3a bytes) 000498[0] 1785394978.898755 trunc from 0x0396b6 to 0x13f99e (0x1062e9 bytes) 000500[0] 1785394978.950225 trunc from 0x13f99f to 0x0c97c7 (0x761d7 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4979 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x544cfb 000007[0] 1785394980.135540 trunc from 00000000 to 0x544cfa (0x544cfb bytes) 000009[0] 1785394980.183114 mapread 0x200000 thru 0x2000a6 (0xa7 bytes) 000011[0] 1785394980.233483 mapread 0x129000 thru 0x12996e (0x96f bytes) 000012[0] 1785394980.242711 write 0x6be000 thru 0x6cb892 (0xd893 bytes) 000013[1] 1785394980.247866 mapread 0x091000 thru 0x098034 (0x7035 bytes) 000015[1] 1785394980.392816 mapread 0x158000 thru 0x161321 (0x9322 bytes) 000016[0] 1785394980.457952 write 0x945000 thru 0x948fa9 (0x3faa bytes) 000018[1] 1785394980.566951 write_OD 0x536000 thru 0x542fff (0xd000 bytes) 000020[1] 1785394980.714978 mapwrite 0x2a6000 thru 0x2b5e39 (0xfe3a bytes) 000022[0] 1785394980.831323 mapwrite 0x0a8000 thru 0x0b3250 (0xb251 bytes) 000023[1] 1785394981.059146 write_OD 0x0f2000 thru 0x0f2fff (0x1000 bytes) 000024[1] 1785394981.264567 mapwrite 0x4f8000 thru 0x4f8bf8 (0xbf9 bytes) 000025[1] 1785394981.443654 read_OD 0x2b4000 thru 0x2bffff (0xc000 bytes) 000026[1] 1785394981.506275 mapwrite 0x940000 thru 0x945c2a (0x5c2b bytes) 000027[0] 1785394981.779947 read 0x251000 thru 0x257101 (0x6102 bytes) 000029[1] 1785394981.920440 trunc from 0x948faa to 0x0d171e (0x87788b bytes) 000030[0] 1785394982.033572 write 0x468000 thru 0x4684c2 (0x4c3 bytes) 000032[0] 1785394982.157130 mapwrite 0x1eb000 thru 0x1ee8fb (0x38fc bytes) 000035[0] 1785394982.529320 trunc from 0x4684c3 to 0x02dd05 (0x43a7bd bytes) 000038[0] 1785394982.740544 mapread 0x009000 thru 0x018faf (0xffb0 bytes) 000044[1] 1785394982.979084 trunc from 0x02dd06 to 0x2a1800 (0x273afb bytes) 000045[1] 1785394983.073365 write_OD 0x194000 thru 0x19dfff (0xa000 bytes) 000051[0] 1785394983.259432 write 0x448000 thru 0x4489b5 (0x9b6 bytes) 000052[0] 1785394983.292904 mapread 0x3e7000 thru 0x3f61d5 (0xf1d6 bytes) 000053[0] 1785394983.419966 mapwrite 0x999000 thru 0x9a5522 (0xc523 bytes) truncating to largest ever: 0x5e7e92 000054[1] 1785394983.708551 trunc from 0x9a5523 to 0x5e7e91 (0x3bd691 bytes) 000055[0] 1785394983.838884 mapwrite 0x525000 thru 0x52a279 (0x527a bytes) 000056[1] 1785394984.061226 trunc from 0x5e7e92 to 0x31ac40 (0x2cd251 bytes) 000057[0] 1785394984.130260 trunc from 0x31ac41 to 0x30dcd0 (0xcf70 bytes) 000058[1] 1785394984.238427 write_OD 0x258000 thru 0x25cfff (0x5000 bytes) 000059[0] 1785394984.366510 write 0x94d000 thru 0x953463 (0x6464 bytes) 000060[1] 1785394984.405412 mapwrite 0x102000 thru 0x109b2f (0x7b30 bytes) 000061[1] 1785394984.651884 mapread 0x1be000 thru 0x1cd891 (0xf892 bytes) 000062[1] 1785394984.688420 mapread 0x0a3000 thru 0x0b151a (0xe51b bytes) 000064[1] 1785394984.778333 write_OD 0x9f7000 thru 0x9fefff (0x8000 bytes) 000066[0] 1785394984.910817 read 0x162000 thru 0x16cbc2 (0xabc3 bytes) 000069[0] 1785394984.935060 mapread 0x9ca000 thru 0x9cecf2 (0x4cf3 bytes) 000070[0] 1785394985.037212 mapwrite 0x657000 thru 0x65b575 (0x4576 bytes) 000071[1] 1785394985.278060 read_OD 0x9e0000 thru 0x9ebfff (0xc000 bytes) 000072[0] 1785394985.364353 write 0x7f8000 thru 0x802b86 (0xab87 bytes) 000073[0] 1785394985.424094 read 0x7a2000 thru 0x7ae4f5 (0xc4f6 bytes) 000074[0] 1785394985.459744 write 0x7b3000 thru 0x7bc18b (0x918c bytes) 000075[0] 1785394985.468688 mapwrite 0x27d000 thru 0x2877e0 (0xa7e1 bytes) truncating to largest ever: 0x76c9b9 000076[1] 1785394985.655884 trunc from 0x9ff000 to 0x76c9b8 (0x292647 bytes) 000079[1] 1785394985.817353 mapwrite 0x2c2000 thru 0x2c638c (0x438d bytes) 000080[1] 1785394986.055559 mapread 0x3b0000 thru 0x3bd8bc (0xd8bd bytes) 000081[1] 1785394986.097354 read_OD 0x4dd000 thru 0x4eafff (0xe000 bytes) truncating to largest ever: 0x80b442 000082[1] 1785394986.150235 trunc from 0x76c9b9 to 0x80b441 (0x9ea89 bytes) 000084[1] 1785394986.295508 write_OD 0x52d000 thru 0x52efff (0x2000 bytes) 000085[0] 1785394986.566042 mapread 0x3ad000 thru 0x3ad63a (0x63b bytes) 000086[1] 1785394986.590813 mapread 0x72b000 thru 0x73390a (0x890b bytes) 000089[1] 1785394986.685517 read_OD 0x631000 thru 0x633fff (0x3000 bytes) 000090[0] 1785394986.695387 read 0x557000 thru 0x55e77d (0x777e bytes) 000091[1] 1785394986.709251 trunc from 0x80b442 to 0x50f975 (0x2fbacc bytes) 000092[1] 1785394986.766130 mapwrite 0x7ea000 thru 0x7f3c26 (0x9c27 bytes) 000093[1] 1785394986.895070 write_OD 0x833000 thru 0x834fff (0x2000 bytes) 000094[0] 1785394986.957977 read 0x0c1000 thru 0x0d0796 (0xf797 bytes) 000096[1] 1785394986.983266 mapwrite 0x89a000 thru 0x8a8d5b (0xed5c bytes) 000097[1] 1785394987.255198 trunc from 0x8a8d5c to 0x3d4115 (0x4d4c46 bytes) 000098[1] 1785394987.323670 write_OD 0x655000 thru 0x655fff (0x1000 bytes) 000102[1] 1785394987.463044 trunc from 0x656000 to 0x0d2aa5 (0x58355a bytes) 000103[1] 1785394987.541553 mapread 0x006000 thru 0x009f76 (0x3f77 bytes) 000106[1] 1785394987.671992 read_OD 0x0cf000 thru 0x0d0fff (0x2000 bytes) 000109[0] 1785394987.736956 read 0x02a000 thru 0x02c801 (0x2802 bytes) truncating to largest ever: 0x8a2f67 000113[1] 1785394987.785218 trunc from 0x0d2aa6 to 0x8a2f66 (0x7d04c1 bytes) 000118[1] 1785394988.071341 mapwrite 0x6fe000 thru 0x7030af (0x50b0 bytes) 000121[0] 1785394988.774724 mapwrite 0x1d3000 thru 0x1d97e7 (0x67e8 bytes) 000122[0] 1785394989.398387 mapwrite 0x3cf000 thru 0x3de07d (0xf07e bytes) 000124[0] 1785394989.549867 mapwrite 0x00e000 thru 0x015ac5 (0x7ac6 bytes) 000125[1] 1785394989.811705 trunc from 0x8a2f67 to 0x12d6bb (0x7758ab bytes) 000127[1] 1785394989.918281 mapwrite 0x0ca000 thru 0x0d0e97 (0x6e98 bytes) 000128[0] 1785394990.068616 mapread 0x0cd000 thru 0x0d9361 (0xc362 bytes) 000129[1] 1785394990.251159 mapwrite 0x084000 thru 0x086df8 (0x2df9 bytes) 000130[0] 1785394990.385520 mapread 0x11d000 thru 0x126a75 (0x9a76 bytes) 000133[1] 1785394990.493360 mapread 0x09b000 thru 0x09cc3a (0x1c3b bytes) 000134[0] 1785394990.509421 mapwrite 0x027000 thru 0x02dc72 (0x6c73 bytes) 000135[1] 1785394990.790608 mapwrite 0x326000 thru 0x32bd88 (0x5d89 bytes) 000137[0] 1785394991.021687 write 0x71c000 thru 0x71f73b (0x373c bytes) 000139[0] 1785394991.084060 read 0x53f000 thru 0x540213 (0x1214 bytes) 000140[1] 1785394991.119225 mapwrite 0x99f000 thru 0x9acf15 (0xdf16 bytes) 000143[0] 1785394991.541121 mapread 0x2cb000 thru 0x2cdb6e (0x2b6f bytes) 000144[0] 1785394991.622171 read 0x7a0000 thru 0x7a4b35 (0x4b36 bytes) 000147[1] 1785394991.643203 read_OD 0x3d3000 thru 0x3d6fff (0x4000 bytes) 000148[0] 1785394991.661905 read 0x13b000 thru 0x143ebc (0x8ebd bytes) 000149[0] 1785394991.684157 read 0x0e1000 thru 0x0eac6a (0x9c6b bytes) 000150[1] 1785394991.724301 read_OD 0x20d000 thru 0x212fff (0x6000 bytes) 000151[1] 1785394991.760316 read_OD 0x846000 thru 0x852fff (0xd000 bytes) 000152[1] 1785394991.806356 read_OD 0x6c4000 thru 0x6ccfff (0x9000 bytes) 000154[0] 1785394991.831346 read 0x72f000 thru 0x737415 (0x8416 bytes) 000155[1] 1785394991.853390 mapread 0x0d5000 thru 0x0d8d6c (0x3d6d bytes) 000157[0] 1785394991.917380 mapread 0x856000 thru 0x85ab0e (0x4b0f bytes) 000158[1] 1785394991.949131 write_OD 0x1b7000 thru 0x1b9fff (0x3000 bytes) 000160[0] 1785394992.093736 mapread 0x902000 thru 0x90a2d5 (0x82d6 bytes) 000162[0] 1785394992.161888 mapread 0x335000 thru 0x335a06 (0xa07 bytes) 000163[0] 1785394992.170175 write 0x093000 thru 0x099be0 (0x6be1 bytes) 000164[0] 1785394992.243048 write 0x3a0000 thru 0x3a92e3 (0x92e4 bytes) 000166[1] 1785394992.311740 mapread 0x159000 thru 0x15b080 (0x2081 bytes) 000167[1] 1785394992.418442 trunc from 0x9acf16 to 0x582f1e (0x429ff7 bytes) 000168[1] 1785394992.554755 mapread 0x4cc000 thru 0x4d7039 (0xb03a bytes) 000169[1] 1785394992.711929 mapwrite 0x5b9000 thru 0x5c74be (0xe4bf bytes) 000170[0] 1785394993.115632 mapread 0x224000 thru 0x22efbf (0xafc0 bytes) 000171[0] 1785394993.277124 write 0x0db000 thru 0x0de699 (0x369a bytes) 000173[1] 1785394993.428458 trunc from 0x5c74bf to 0x323663 (0x2a3e5b bytes) 000174[1] 1785394993.671195 trunc from 0x323664 to 0x2b908c (0x6a5d7 bytes) 000177[0] 1785394993.747601 trunc from 0x2b908d to 0x4142fb (0x15b26f bytes) 000179[1] 1785394993.825353 mapread 0x126000 thru 0x126842 (0x843 bytes) 000181[1] 1785394993.869122 mapread 0x25f000 thru 0x260b70 (0x1b71 bytes) 000182[1] 1785394993.884618 trunc from 0x4142fc to 0x300398 (0x113f63 bytes) 000183[1] 1785394993.940964 write_OD 0x17d000 thru 0x182fff (0x6000 bytes) 000184[0] 1785394994.051282 mapread 0x1f3000 thru 0x1fca79 (0x9a7a bytes) 000186[1] 1785394994.102757 write_OD 0x6b9000 thru 0x6bffff (0x7000 bytes) 000187[0] 1785394994.213505 mapwrite 0x8bd000 thru 0x8ca9d9 (0xd9da bytes) 000188[0] 1785394994.531150 read 0x6d2000 thru 0x6d4e4e (0x2e4f bytes) 000190[0] 1785394994.589263 trunc from 0x8ca9da to 0x13508b (0x79594e bytes) 000191[1] 1785394994.643020 mapread 0x106000 thru 0x10c7fa (0x67fb bytes) 000192[0] 1785394994.777896 mapwrite 0x35e000 thru 0x35fccf (0x1cd0 bytes) 000193[1] 1785394995.050100 mapread 0x118000 thru 0x11c736 (0x4737 bytes) 000196[0] 1785394995.173639 read 0x121000 thru 0x12d1ef (0xc1f0 bytes) 000197[0] 1785394995.435621 mapwrite 0x91f000 thru 0x92b7fb (0xc7fc bytes) 000200[1] 1785394995.925177 read_OD 0x4d2000 thru 0x4d9fff (0x8000 bytes) 000201[0] 1785394996.049600 trunc from 0x92b7fc to 0x5818ba (0x3a9f41 bytes) 000203[1] 1785394996.191760 trunc from 0x5818bb to 0x840e57 (0x2bf59d bytes) 000207[1] 1785394996.361833 trunc from 0x840e58 to 0x416f11 (0x429f46 bytes) 000208[1] 1785394996.446891 mapread 0x2a0000 thru 0x2a65c9 (0x65ca bytes) 000211[1] 1785394996.594663 mapread 0x1e9000 thru 0x1f8de0 (0xfde1 bytes) 000213[1] 1785394996.706695 write_OD 0x416000 thru 0x420fff (0xb000 bytes) 000214[0] 1785394996.871882 mapwrite 0x148000 thru 0x14960e (0x160f bytes) 000215[0] 1785394997.018592 read 0x2f5000 thru 0x2f93c8 (0x43c9 bytes) 000216[1] 1785394997.030253 mapwrite 0x905000 thru 0x90cecb (0x7ecc bytes) 000217[0] 1785394997.333485 read 0x64a000 thru 0x655e19 (0xbe1a bytes) 000218[0] 1785394997.430221 write 0x17b000 thru 0x17e211 (0x3212 bytes) 000221[0] 1785394997.667499 trunc from 0x90cecc to 0x02fd0c (0x8dd1bf bytes) 000222[0] 1785394997.720661 mapwrite 0x3fc000 thru 0x3fd3a4 (0x13a5 bytes) 000224[0] 1785394997.927300 read 0x2e4000 thru 0x2f09ff (0xca00 bytes) 000228[1] 1785394998.094848 mapread 0x11b000 thru 0x11c648 (0x1649 bytes) 000229[1] 1785394998.129078 mapread 0x1d5000 thru 0x1e01cb (0xb1cc bytes) 000234[1] 1785394998.153995 write_OD 0x1f8000 thru 0x1fefff (0x7000 bytes) 000235[1] 1785394998.223358 mapread 0x1c5000 thru 0x1d1e62 (0xce63 bytes) 000237[0] 1785394998.256144 trunc from 0x3fd3a5 to 0x095d8f (0x367615 bytes) 000239[1] 1785394998.383490 mapread 0x06d000 thru 0x06df36 (0xf37 bytes) 000240[1] 1785394998.453021 write_OD 0x9ea000 thru 0x9f2fff (0x9000 bytes) 000242[0] 1785394998.609971 write 0x85d000 thru 0x863b6e (0x6b6f bytes) 000243[1] 1785394998.726854 mapread 0x5c4000 thru 0x5d013c (0xc13d bytes) 000244[0] 1785394998.913619 write 0x8c9000 thru 0x8d7e22 (0xee23 bytes) 000246[0] 1785394999.122418 mapwrite 0x454000 thru 0x45667a (0x267b bytes) 000247[0] 1785394999.402277 mapread 0x3e5000 thru 0x3e5ca0 (0xca1 bytes) 000249[1] 1785394999.522400 read_OD 0x442000 thru 0x445fff (0x4000 bytes) 000250[1] 1785394999.565896 write_OD 0x918000 thru 0x921fff (0xa000 bytes) 000251[1] 1785394999.664658 mapread 0x063000 thru 0x066cf9 (0x3cfa bytes) 000253[0] 1785394999.766379 write 0x5e4000 thru 0x5ee998 (0xa999 bytes) 000255[0] 1785394999.806362 mapwrite 0x6bc000 thru 0x6bd469 (0x146a bytes) 000260[1] 1785395000.089696 trunc from 0x9f3000 to 0x1d03ae (0x822c51 bytes) 000261[0] 1785395000.223155 mapread 0x0c0000 thru 0x0cc8ac (0xc8ad bytes) 000264[0] 1785395000.470657 mapwrite 0x0ab000 thru 0x0b1d8f (0x6d90 bytes) 000268[1] 1785395000.850930 trunc from 0x1d03af to 0x5f9867 (0x4294b9 bytes) 000269[1] 1785395000.914863 mapwrite 0x530000 thru 0x53e534 (0xe535 bytes) 000270[0] 1785395001.172847 mapread 0x1b4000 thru 0x1c1ec4 (0xdec5 bytes) 000272[1] 1785395001.492510 read_OD 0x20f000 thru 0x218fff (0xa000 bytes) 000273[1] 1785395001.527198 mapwrite 0x950000 thru 0x95e59a (0xe59b bytes) 000277[1] 1785395001.849438 mapwrite 0x69a000 thru 0x69e324 (0x4325 bytes) 000278[1] 1785395001.937163 trunc from 0x95e59b to 0x5a6642 (0x3b7f58 bytes) 000282[0] 1785395002.050643 write 0x77b000 thru 0x77b73b (0x73c bytes) 000284[1] 1785395002.116690 trunc from 0x77b73c to 0x07a44f (0x7012ec bytes) 000287[0] 1785395002.204121 write 0x54c000 thru 0x558f54 (0xcf55 bytes) 000291[1] 1785395002.297340 trunc from 0x558f55 to 0x1ae0ec (0x3aae68 bytes) 000292[0] 1785395002.354696 mapread 0x114000 thru 0x12346f (0xf470 bytes) 000293[1] 1785395002.492696 trunc from 0x1ae0ed to 0x6d3ec7 (0x525ddb bytes) 000298[1] 1785395002.576584 read_OD 0x0be000 thru 0x0c8fff (0xb000 bytes) 000300[0] 1785395002.594813 mapwrite 0x2b6000 thru 0x2bc8e1 (0x68e2 bytes) 000306[1] 1785395002.742040 mapread 0x564000 thru 0x56cf4a (0x8f4b bytes) 000308[0] 1785395002.857173 write 0x2e8000 thru 0x2f433b (0xc33c bytes) 000310[1] 1785395002.965161 trunc from 0x6d3ec8 to 0x34dc86 (0x386241 bytes) 000311[1] 1785395003.054546 read_OD 0x139000 thru 0x147fff (0xf000 bytes) 000312[1] 1785395003.132150 write_OD 0x3ac000 thru 0x3aefff (0x3000 bytes) 000316[0] 1785395003.268908 mapread 0x058000 thru 0x0585c9 (0x5ca bytes) 000318[1] 1785395003.281771 mapread 0x180000 thru 0x18a92f (0xa930 bytes) 000320[1] 1785395003.378816 write_OD 0x366000 thru 0x36bfff (0x6000 bytes) 000325[1] 1785395003.514750 mapread 0x053000 thru 0x055d2a (0x2d2b bytes) 000326[1] 1785395003.543589 mapread 0x0a7000 thru 0x0a8b6f (0x1b70 bytes) 000327[0] 1785395003.575528 read 0x292000 thru 0x29e3ff (0xc400 bytes) 000328[1] 1785395003.682578 read_OD 0x0f7000 thru 0x102fff (0xc000 bytes) 000329[1] 1785395003.707303 mapread 0x1b2000 thru 0x1ba269 (0x826a bytes) truncating to largest ever: 0x91b7cc 000337[0] 1785395003.861594 trunc from 0x3af000 to 0x91b7cb (0x56c7cc bytes) 000338[1] 1785395003.944341 trunc from 0x91b7cc to 0x653cf2 (0x2c7ad9 bytes) 000339[0] 1785395003.992279 write 0x66d000 thru 0x67bec0 (0xeec1 bytes) 000340[1] 1785395004.050463 mapwrite 0x993000 thru 0x999392 (0x6393 bytes) 000341[1] 1785395004.473135 trunc from 0x999393 to 0x11dbcd (0x87b7c5 bytes) 000346[0] 1785395004.591403 read 0x034000 thru 0x03897d (0x497e bytes) 000348[0] 1785395004.725269 mapwrite 0x0c4000 thru 0x0ca688 (0x6689 bytes) 000350[0] 1785395004.991030 read 0x01e000 thru 0x027f87 (0x9f88 bytes) 000351[0] 1785395005.058980 read 0x0d6000 thru 0x0da11d (0x411e bytes) 000354[1] 1785395005.063346 mapread 0x0dd000 thru 0x0e8c4d (0xbc4e bytes) 000357[0] 1785395005.219504 read 0x055000 thru 0x06111b (0xc11c bytes) 000358[0] 1785395005.274121 mapread 0x10b000 thru 0x10ed14 (0x3d15 bytes) 000361[0] 1785395005.277663 mapread 0x00f000 thru 0x01e50c (0xf50d bytes) 000362[0] 1785395005.281281 mapwrite 0x9f3000 thru 0x9ff51d (0xc51e bytes) 000363[1] 1785395005.760157 mapread 00000000 thru 0x007b99 (0x7b9a bytes) 000365[0] 1785395005.982430 write 0x926000 thru 0x928731 (0x2732 bytes) 000367[0] 1785395006.122268 write 0x653000 thru 0x656727 (0x3728 bytes) 000368[0] 1785395006.188483 mapread 0x130000 thru 0x1324b8 (0x24b9 bytes) 000370[0] 1785395006.362530 trunc from 0x9ff51e to 0x7556bf (0x2a9e5e bytes) 000373[0] 1785395006.527890 mapwrite 0x30a000 thru 0x3141dd (0xa1de bytes) 000376[1] 1785395006.937896 trunc from 0x7556c0 to 0x0d2fe6 (0x6826d9 bytes) 000377[1] 1785395007.048960 write_OD 0x246000 thru 0x249fff (0x4000 bytes) 000382[0] 1785395007.253510 read 0x247000 thru 0x24978a (0x278b bytes) 000384[0] 1785395007.477888 mapread 0x0b6000 thru 0x0c478a (0xe78b bytes) 000385[1] 1785395007.481086 read_OD 0x058000 thru 0x064fff (0xd000 bytes) 000394[0] 1785395007.554283 write 0x76a000 thru 0x76d318 (0x3319 bytes) 000396[0] 1785395007.686654 mapwrite 0x9ef000 thru 0x9ef023 (0x24 bytes) 000397[1] 1785395008.043534 mapread 0x8b8000 thru 0x8ba9fc (0x29fd bytes) 000398[0] 1785395008.451173 mapread 0x6dd000 thru 0x6e9fa6 (0xcfa7 bytes) 000400[0] 1785395008.571812 mapwrite 0x79d000 thru 0x7a44b8 (0x74b9 bytes) 000402[1] 1785395009.020929 mapread 0x8c4000 thru 0x8c456f (0x570 bytes) 000404[1] 1785395009.111375 mapread 0x899000 thru 0x89dc2b (0x4c2c bytes) 000409[0] 1785395009.241999 mapread 0x8aa000 thru 0x8b3a41 (0x9a42 bytes) 000412[0] 1785395009.374693 mapread 0x5ec000 thru 0x5f49da (0x89db bytes) 000415[1] 1785395009.520582 read_OD 0x385000 thru 0x388fff (0x4000 bytes) 000416[0] 1785395009.533443 mapread 0x15a000 thru 0x15e4fd (0x44fe bytes) 000421[1] 1785395009.615602 write_OD 0x280000 thru 0x280fff (0x1000 bytes) 000422[1] 1785395009.738811 mapread 0x301000 thru 0x3083ef (0x73f0 bytes) 000424[0] 1785395009.811037 read 0x839000 thru 0x846a53 (0xda54 bytes) 000425[1] 1785395009.847794 mapread 0x267000 thru 0x26db15 (0x6b16 bytes) 000429[0] 1785395009.866569 read 0x7bd000 thru 0x7c78e1 (0xa8e2 bytes) 000430[1] 1785395009.884524 trunc from 0x9ef024 to 0x25a588 (0x794a9b bytes) 000438[1] 1785395010.207211 write_OD 0x054000 thru 0x061fff (0xe000 bytes) 000439[1] 1785395010.359360 trunc from 0x25a589 to 0x4e6263 (0x28bcdb bytes) 000440[1] 1785395010.441362 mapread 0x271000 thru 0x279ef3 (0x8ef4 bytes) 000441[1] 1785395010.497292 mapread 0x25e000 thru 0x264946 (0x6947 bytes) 000445[0] 1785395010.624442 read 0x2ee000 thru 0x2fb879 (0xd87a bytes) 000446[0] 1785395010.670301 mapwrite 0x0ab000 thru 0x0adea0 (0x2ea1 bytes) 000448[0] 1785395010.831982 write 0x04b000 thru 0x04c362 (0x1363 bytes) 000449[0] 1785395010.904624 trunc from 0x4e6264 to 0x543349 (0x5d0e6 bytes) 000450[0] 1785395010.974751 write 0x3b7000 thru 0x3be633 (0x7634 bytes) 000451[0] 1785395010.993604 read 0x49e000 thru 0x4ac815 (0xe816 bytes) 000452[1] 1785395011.052062 mapread 0x4a6000 thru 0x4ab510 (0x5511 bytes) 000453[0] 1785395011.249669 mapread 0x43a000 thru 0x43dd3b (0x3d3c bytes) 000454[1] 1785395011.293562 trunc from 0x54334a to 0x2951af (0x2ae19a bytes) 000455[0] 1785395011.372057 write 0x27e000 thru 0x286790 (0x8791 bytes) 000456[1] 1785395011.441285 trunc from 0x2951b0 to 0x831449 (0x59c29a bytes) 000458[1] 1785395011.551676 trunc from 0x83144a to 0x3a8431 (0x489018 bytes) 000460[0] 1785395011.617607 mapread 0x19d000 thru 0x1a8a80 (0xba81 bytes) 000461[1] 1785395011.700580 mapwrite 0x0b1000 thru 0x0b4cac (0x3cad bytes) 000462[1] 1785395011.892943 trunc from 0x3a8432 to 0x40655c (0x5e12b bytes) 000464[0] 1785395011.993908 read 0x0a9000 thru 0x0ae8c0 (0x58c1 bytes) 000467[1] 1785395012.018372 mapread 0x18d000 thru 0x190ae5 (0x3ae6 bytes) 000469[1] 1785395012.026314 mapread 0x3c1000 thru 0x3c6851 (0x5852 bytes) 000470[1] 1785395012.055318 read_OD 0x055000 thru 0x05ffff (0xb000 bytes) 000471[1] 1785395012.069850 mapwrite 0x901000 thru 0x9031d4 (0x21d5 bytes) 000472[1] 1785395012.171246 mapread 0x3cd000 thru 0x3d7cd8 (0xacd9 bytes) 000473[1] 1785395012.200663 mapread 0x366000 thru 0x36ba50 (0x5a51 bytes) 000475[1] 1785395012.214143 mapread 0x179000 thru 0x180b7e (0x7b7f bytes) 000480[1] 1785395012.272075 trunc from 0x9031d5 to 0x1523d8 (0x7b0dfc bytes) 000481[1] 1785395012.316800 mapread 0x038000 thru 0x045ad3 (0xdad4 bytes) 000482[1] 1785395012.369944 read_OD 0x0de000 thru 0x0e7fff (0xa000 bytes) 000484[0] 1785395012.409423 trunc from 0x1523d9 to 0x685a58 (0x533680 bytes) 000486[0] 1785395012.480182 trunc from 0x685a59 to 0x446720 (0x23f338 bytes) 000487[0] 1785395012.496531 read 0x025000 thru 0x02678d (0x178e bytes) 000488[0] 1785395012.501426 mapread 0x3fe000 thru 0x409ae7 (0xbae8 bytes) 000489[0] 1785395012.542400 read 0x27f000 thru 0x28d2c2 (0xe2c3 bytes) 000490[0] 1785395012.550439 mapwrite 0x831000 thru 0x8383ce (0x73cf bytes) 000491[0] 1785395012.741889 mapread 0x406000 thru 0x410c31 (0xac32 bytes) 000495[1] 1785395012.860232 write_OD 0x608000 thru 0x60afff (0x3000 bytes) 000496[1] 1785395013.004461 mapread 0x067000 thru 0x06ec5f (0x7c60 bytes) 000497[1] 1785395013.129254 mapwrite 0x7d1000 thru 0x7d5d9f (0x4da0 bytes) All operations completed A-OK! PASS 16a (88s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 03:03:40 (1785395020) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.940989 s, 8.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.533626 s, 15.7 MB/s Chance of close/open is 1 in 50 Seed set to 5024 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005[1] 1785395025.754898 mapwrite 0x158aeb thru 0x15a1af (0x16c5 bytes) truncating to largest ever: 0x90a24d 000006[0] 1785395025.903026 trunc from 0x15a1b0 to 0x90a24c (0x7b009d bytes) 000007[0] 1785395025.950255 mapread 0x6cd99b thru 0x6cea0e (0x1074 bytes) 000010[0] 1785395026.015199 trunc from 0x90a24d to 0x728004 (0x1e2248 bytes) 000015[1] 1785395026.217843 mapread 0x11380f thru 0x115718 (0x1f0a bytes) 000016[0] 1785395026.341598 trunc from 0x728005 to 0x05a100 (0x6cdf04 bytes) 000017[1] 1785395026.478763 trunc from 0x05a101 to 0x5d81dd (0x57e0dd bytes) truncating to largest ever: 0x9f1c5a 000019[1] 1785395026.683562 trunc from 0x5d81de to 0x9f1c59 (0x419a7c bytes) 000020[0] 1785395026.799989 trunc from 0x9f1c5a to 0x7384e5 (0x2b9774 bytes) 000021[1] 1785395026.969213 mapread 0x02e7b9 thru 0x030419 (0x1c61 bytes) 000022[1] 1785395027.117655 mapread 0x663653 thru 0x664955 (0x1303 bytes) 000024[1] 1785395027.227098 read 0x5917ff thru 0x5929a3 (0x11a5 bytes) 000026[0] 1785395027.242742 write 0x6d0c78 thru 0x6d0e90 (0x219 bytes) 000027[0] 1785395027.358120 read 0x449c35 thru 0x44b3ba (0x1786 bytes) 000028[0] 1785395027.424937 trunc from 0x7384e6 to 0x432740 (0x305da5 bytes) 000029[0] 1785395027.601288 mapwrite 0x0fa743 thru 0x0fbf2d (0x17eb bytes) 000034[0] 1785395028.020995 trunc from 0x432741 to 0x47c472 (0x49d32 bytes) 000035[0] 1785395028.080555 read 0x28124d thru 0x2831a0 (0x1f54 bytes) 000036[0] 1785395028.186214 mapread 0x1c6a3d thru 0x1c8720 (0x1ce4 bytes) 000041[1] 1785395028.334149 write 0x8b51c8 thru 0x8b62c5 (0x10fe bytes) 000042[0] 1785395028.386221 mapread 0x6238ea thru 0x624974 (0x108b bytes) 000044[1] 1785395028.545553 write 0x94a2e2 thru 0x94b73f (0x145e bytes) 000048[0] 1785395028.600771 trunc from 0x94b740 to 0x775133 (0x1d660c bytes) 000051[0] 1785395028.664421 read 0x1ed4d6 thru 0x1ee4ca (0xff5 bytes) 000054[1] 1785395028.731367 mapread 0x2c59b9 thru 0x2c6385 (0x9cd bytes) 000055[1] 1785395028.825651 read 0x29cd65 thru 0x29e725 (0x19c1 bytes) 000056[1] 1785395028.855461 read 0x1120b3 thru 0x113690 (0x15de bytes) 000057[1] 1785395029.009056 mapwrite 0x7de8bb thru 0x7df4e0 (0xc26 bytes) 000059[1] 1785395029.623869 mapread 0x305161 thru 0x305813 (0x6b3 bytes) 000062[0] 1785395029.723758 read 0x5b9ac6 thru 0x5b9c96 (0x1d1 bytes) 000064[1] 1785395029.821619 mapwrite 0x69e91d thru 0x69f933 (0x1017 bytes) 000066[1] 1785395030.072136 read 0x01ea1b thru 0x020300 (0x18e6 bytes) 000067[1] 1785395030.111407 mapread 0x764609 thru 0x764a1e (0x416 bytes) 000068[1] 1785395030.173413 mapwrite 0x0cf445 thru 0x0d0295 (0xe51 bytes) 000069[1] 1785395030.419138 write 0x3dfade thru 0x3dfeec (0x40f bytes) 000070[0] 1785395030.540200 write 0x69f29a thru 0x6a09d1 (0x1738 bytes) 000072[0] 1785395030.735250 trunc from 0x7df4e1 to 0x251d63 (0x58d77d bytes) 000073[0] 1785395030.804831 write 0x1a4c31 thru 0x1a5a7c (0xe4c bytes) 000074[1] 1785395030.914817 write 0x27dc8f thru 0x27e536 (0x8a8 bytes) 000075[0] 1785395031.013112 trunc from 0x27e537 to 0x31a9d9 (0x9c4a3 bytes) 000078[1] 1785395031.115822 read 0x24620e thru 0x2474f5 (0x12e8 bytes) 000079[0] 1785395031.403338 mapwrite 0x139669 thru 0x13a6f3 (0x108b bytes) 000080[1] 1785395031.689623 mapwrite 0x03548a thru 0x035a36 (0x5ad bytes) 000081[1] 1785395032.089274 mapwrite 0x046f26 thru 0x0471c1 (0x29c bytes) 000082[0] 1785395032.358883 write 0x7f6dae thru 0x7f8a94 (0x1ce7 bytes) 000083[0] 1785395032.426468 trunc from 0x7f8a95 to 0x1b8aa6 (0x63ffee bytes) 000084[1] 1785395032.483902 mapread 0x0ed3a6 thru 0x0eef52 (0x1bad bytes) 000085[1] 1785395032.589123 trunc from 0x1b8aa7 to 0x6969ab (0x4ddf05 bytes) 000086[0] 1785395032.722023 write 0x2cd792 thru 0x2cf0fe (0x196d bytes) 000087[1] 1785395032.810457 read 0x5cdf60 thru 0x5cf61b (0x16bc bytes) 000088[1] 1785395032.872678 mapwrite 0x71d389 thru 0x71e1cd (0xe45 bytes) 000091[1] 1785395033.310625 mapwrite 0x6814ac thru 0x6828d6 (0x142b bytes) 000092[0] 1785395033.486436 write 0x7e4648 thru 0x7e4f5b (0x914 bytes) 000093[0] 1785395033.568325 mapwrite 0x2f3a42 thru 0x2f4dff (0x13be bytes) 000094[1] 1785395033.752919 mapwrite 0x9d64ca thru 0x9d6d65 (0x89c bytes) 000095[0] 1785395034.025506 mapwrite 0x9690c1 thru 0x96a679 (0x15b9 bytes) 000096[1] 1785395034.296762 write 0x086ef0 thru 0x0880d5 (0x11e6 bytes) 000098[1] 1785395034.406545 trunc from 0x9d6d66 to 0x12d58f (0x8a97d6 bytes) 000099[0] 1785395034.530762 write 0x66b415 thru 0x66b867 (0x453 bytes) 000101[1] 1785395034.663564 read 0x5d029c thru 0x5d1bba (0x191f bytes) 000104[1] 1785395034.792003 read 0x305fc3 thru 0x307f44 (0x1f82 bytes) 000107[0] 1785395034.880296 write 0x0db343 thru 0x0dbbae (0x86c bytes) 000108[1] 1785395034.927502 write 0x638d46 thru 0x63aa6e (0x1d29 bytes) 000109[1] 1785395035.007624 read 0x6361cf thru 0x636c4e (0xa80 bytes) 000110[1] 1785395035.041587 write 0x86bd08 thru 0x86cc5b (0xf54 bytes) 000112[0] 1785395035.067975 mapread 0x5eaf6c thru 0x5ec11d (0x11b2 bytes) 000114[0] 1785395035.198912 read 0x5c3f8a thru 0x5c52b8 (0x132f bytes) 000117[1] 1785395035.292036 trunc from 0x86cc5c to 0x19caec (0x6d016f bytes) 000121[0] 1785395035.463529 trunc from 0x19caed to 0x928372 (0x78b886 bytes) 000122[0] 1785395035.575121 mapread 0x1f07cb thru 0x1f13d2 (0xc08 bytes) 000125[1] 1785395035.700518 write 0x1c5d01 thru 0x1c5d33 (0x33 bytes) 000126[0] 1785395035.775711 trunc from 0x928373 to 0x12eab3 (0x7f98bf bytes) 000130[1] 1785395035.852495 read 0x05452d thru 0x0551b3 (0xc87 bytes) 000131[1] 1785395035.917628 mapwrite 0x7debd5 thru 0x7e04df (0x190b bytes) 000132[1] 1785395036.239544 mapread 0x6a8985 thru 0x6a9969 (0xfe5 bytes) 000133[1] 1785395036.256619 trunc from 0x7e04e0 to 0x5fe71f (0x1e1dc0 bytes) 000136[1] 1785395036.432208 read 0x340b90 thru 0x341e4a (0x12bb bytes) 000139[0] 1785395036.618816 write 0x8f974c thru 0x8fb492 (0x1d47 bytes) 000140[1] 1785395036.649811 read 0x01140c thru 0x013180 (0x1d75 bytes) 000142[1] 1785395036.673375 trunc from 0x8fb493 to 0x7d8127 (0x12336b bytes) 000145[1] 1785395036.742033 trunc from 0x7d8128 to 0x478ee7 (0x35f240 bytes) 000150[1] 1785395036.792289 mapread 0x0ee9ba thru 0x0eefc7 (0x60e bytes) 000156[1] 1785395036.877147 write 0x9e5059 thru 0x9e603a (0xfe2 bytes) 000157[1] 1785395036.884144 write 0x237b67 thru 0x2383e6 (0x880 bytes) 000162[0] 1785395036.933405 mapread 0x16f40b thru 0x1707a1 (0x1397 bytes) 000163[1] 1785395036.974540 mapread 0x902dd4 thru 0x90494b (0x1b78 bytes) 000165[0] 1785395037.013295 mapread 0x5471b7 thru 0x548212 (0x105c bytes) 000166[0] 1785395037.041965 read 0x04b96d thru 0x04d066 (0x16fa bytes) 000169[0] 1785395037.048541 mapwrite 0x32500b thru 0x326978 (0x196e bytes) 000172[0] 1785395037.197530 mapwrite 0x686ee1 thru 0x687b5a (0xc7a bytes) 000173[0] 1785395037.449036 mapwrite 0x74483e thru 0x745e9a (0x165d bytes) 000176[0] 1785395037.652529 write 0x05ef73 thru 0x05f530 (0x5be bytes) 000181[1] 1785395037.854504 mapwrite 0x5fbf49 thru 0x5fd37d (0x1435 bytes) 000184[0] 1785395038.180919 write 0x1a272d thru 0x1a404b (0x191f bytes) 000187[0] 1785395038.239811 trunc from 0x9e603b to 0x5c9b78 (0x41c4c2 bytes) 000188[1] 1785395038.323695 trunc from 0x5c9b79 to 0x4aba2f (0x11e149 bytes) 000190[1] 1785395038.396506 read 0x202f39 thru 0x204301 (0x13c9 bytes) 000191[0] 1785395038.405307 write 0x8f7104 thru 0x8f8598 (0x1495 bytes) 000195[1] 1785395038.503522 mapread 0x05b58d thru 0x05bb44 (0x5b8 bytes) 000201[0] 1785395038.674351 read 0x26476e thru 0x2650a6 (0x939 bytes) 000202[1] 1785395038.750597 mapread 0x51a670 thru 0x51ae36 (0x7c7 bytes) 000203[1] 1785395038.756729 read 0x876cbd thru 0x877f6d (0x12b1 bytes) 000205[0] 1785395038.798804 read 0x887c62 thru 0x88889a (0xc39 bytes) 000207[0] 1785395038.842221 read 0x6f869e thru 0x6fa2cd (0x1c30 bytes) 000210[1] 1785395038.934817 write 0x50abdb thru 0x50b08a (0x4b0 bytes) 000211[1] 1785395038.985419 write 0x08355d thru 0x0836a4 (0x148 bytes) 000212[0] 1785395039.129759 trunc from 0x8f8599 to 0x688e3c (0x26f75c bytes) 000213[0] 1785395039.281043 trunc from 0x688e3d to 0x21fb84 (0x4692b8 bytes) 000214[1] 1785395039.339910 write 0x87cc2c thru 0x87da51 (0xe26 bytes) 000216[1] 1785395039.398476 write 0x1f6368 thru 0x1f72c1 (0xf5a bytes) 000217[0] 1785395039.439215 mapread 0x30b5f6 thru 0x30d269 (0x1c74 bytes) 000219[0] 1785395039.563207 read 0x496ab0 thru 0x497fba (0x150b bytes) 000220[0] 1785395039.576120 mapread 0x01b864 thru 0x01d25d (0x19fa bytes) 000221[1] 1785395039.643436 mapwrite 0x034e3a thru 0x036a00 (0x1bc7 bytes) 000222[1] 1785395039.751757 write 0x19e3a3 thru 0x19f137 (0xd95 bytes) 000224[0] 1785395039.787364 mapread 0x0e57e3 thru 0x0e5c9c (0x4ba bytes) 000225[0] 1785395039.829067 read 0x5767dd thru 0x5777ee (0x1012 bytes) 000226[1] 1785395039.842017 trunc from 0x87da52 to 0x7f19ae (0x8c0a3 bytes) 000229[0] 1785395039.910241 mapwrite 0x6821ff thru 0x683b44 (0x1946 bytes) 000230[0] 1785395040.199484 trunc from 0x7f19af to 0x409ac7 (0x3e7ee7 bytes) 000231[1] 1785395040.333416 mapread 0x12b83e thru 0x12be84 (0x647 bytes) 000240[1] 1785395040.482254 trunc from 0x409ac8 to 0x9d6da6 (0x5cd2df bytes) 000241[0] 1785395040.566682 mapwrite 0x714ab4 thru 0x715d22 (0x126f bytes) 000242[1] 1785395040.768151 mapread 0x886242 thru 0x886dbf (0xb7e bytes) 000245[1] 1785395040.959318 trunc from 0x9d6da7 to 0x9e25d1 (0xb82b bytes) 000246[0] 1785395041.161004 write 0x989a66 thru 0x98b7a1 (0x1d3c bytes) 000250[1] 1785395041.319416 read 0x48d18f thru 0x48d19a (0xc bytes) 000251[1] 1785395041.343929 write 0x99381c thru 0x994b35 (0x131a bytes) 000255[0] 1785395041.467301 mapwrite 0x237e79 thru 0x238b50 (0xcd8 bytes) 000256[1] 1785395041.740769 write 0x175b1c thru 0x1763ed (0x8d2 bytes) 000258[0] 1785395041.879669 mapread 0x16ed03 thru 0x17081a (0x1b18 bytes) 000259[1] 1785395042.011062 mapwrite 0x7ac591 thru 0x7ad548 (0xfb8 bytes) 000263[1] 1785395042.171584 trunc from 0x9e25d2 to 0x2cc099 (0x716538 bytes) 000264[0] 1785395042.254774 mapwrite 0x44e8a8 thru 0x45018e (0x18e7 bytes) 000269[1] 1785395042.662432 read 0x166e6e thru 0x16804c (0x11df bytes) 000270[1] 1785395042.711542 mapwrite 0x6a6427 thru 0x6a7bcc (0x17a6 bytes) 000272[1] 1785395042.906388 write 0x79aa18 thru 0x79b07f (0x668 bytes) 000274[1] 1785395042.912366 read 0x151ddc thru 0x152f24 (0x1149 bytes) 000279[1] 1785395043.185682 read 0x76752d thru 0x7677ae (0x282 bytes) 000281[0] 1785395043.234136 write 0x91e046 thru 0x91f1fc (0x11b7 bytes) 000283[1] 1785395043.347684 read 0x7543bd thru 0x754f54 (0xb98 bytes) 000285[0] 1785395043.395708 trunc from 0x91f1fd to 0x0d07c8 (0x84ea34 bytes) 000287[0] 1785395043.492668 mapread 0x0399d7 thru 0x03a6a9 (0xcd3 bytes) 000289[0] 1785395043.524205 read 0x099d46 thru 0x099e33 (0xee bytes) 000291[1] 1785395043.560396 trunc from 0x0d07c9 to 0x3a0ca6 (0x2d04de bytes) 000292[1] 1785395043.628380 trunc from 0x3a0ca7 to 0x040bb9 (0x3600ed bytes) 000295[0] 1785395043.691597 write 0x16d67f thru 0x16d77a (0xfc bytes) 000297[1] 1785395043.725540 mapwrite 0x84004a thru 0x841015 (0xfcc bytes) 000302[0] 1785395043.841714 write 0x00a15a thru 0x00b757 (0x15fe bytes) 000304[0] 1785395043.932504 trunc from 0x841016 to 0x08df54 (0x7b30c1 bytes) 000307[1] 1785395043.992722 mapread 0x0229be thru 0x0233a4 (0x9e7 bytes) 000310[1] 1785395044.056911 read 0x03180e thru 0x0333bd (0x1bb0 bytes) 000312[1] 1785395044.091663 write 0x42a676 thru 0x42b7eb (0x1176 bytes) 000314[0] 1785395044.132971 mapread 0x003f8c thru 0x005ade (0x1b53 bytes) 000315[0] 1785395044.172638 mapwrite 0x057001 thru 0x057b9d (0xb9d bytes) 000318[0] 1785395044.454917 trunc from 0x42b7ec to 0x9b9759 (0x58df6e bytes) 000320[1] 1785395044.646706 mapwrite 0x4db9da thru 0x4dd6a0 (0x1cc7 bytes) 000321[0] 1785395044.950410 mapread 0x27934c thru 0x27995e (0x613 bytes) 000327[0] 1785395045.151253 write 0x089ce1 thru 0x08a519 (0x839 bytes) 000328[0] 1785395045.210518 mapwrite 0x429aa3 thru 0x42a38a (0x8e8 bytes) 000329[1] 1785395045.367295 mapread 0x2d4719 thru 0x2d4983 (0x26b bytes) 000331[1] 1785395045.452538 write 0x272468 thru 0x2742a7 (0x1e40 bytes) 000333[0] 1785395045.497686 write 0x618081 thru 0x61837d (0x2fd bytes) 000335[1] 1785395045.526764 mapwrite 0x99a2f7 thru 0x99a52b (0x235 bytes) 000342[1] 1785395045.846338 mapwrite 0x8059ab thru 0x806c89 (0x12df bytes) 000343[0] 1785395045.981652 trunc from 0x9b975a to 0x12868c (0x8910cd bytes) 000344[1] 1785395046.117245 mapwrite 0x28e66e thru 0x28feb2 (0x1845 bytes) 000346[0] 1785395046.329658 mapwrite 0x728ecd thru 0x72aec2 (0x1ff6 bytes) 000347[0] 1785395046.624165 mapwrite 0x83846a thru 0x83849a (0x31 bytes) 000348[0] 1785395046.940617 mapwrite 0x7a1a9d thru 0x7a258b (0xaef bytes) 000350[1] 1785395047.271321 mapread 0x7df8ba thru 0x7e1187 (0x18ce bytes) 000351[0] 1785395047.491458 write 0x5f4b5a thru 0x5f5951 (0xdf8 bytes) 000353[0] 1785395047.647995 mapwrite 0x00dfd7 thru 0x00f238 (0x1262 bytes) 000355[1] 1785395047.722560 mapread 0x4ad9e1 thru 0x4adb13 (0x133 bytes) 000358[1] 1785395047.849398 mapwrite 0x24abd3 thru 0x24c07c (0x14aa bytes) 000359[1] 1785395048.136436 trunc from 0x83849b to 0x51a9dd (0x31dabd bytes) 000361[0] 1785395048.260899 write 0x33943d thru 0x33ab4e (0x1712 bytes) 000364[0] 1785395048.434417 read 0x2d551c thru 0x2d5b0b (0x5f0 bytes) 000365[1] 1785395048.501490 read 0x2276e5 thru 0x228dc1 (0x16dd bytes) 000366[1] 1785395048.550139 mapread 0x43420c thru 0x435bce (0x19c3 bytes) 000367[1] 1785395048.617434 write 0x8e67ae thru 0x8e76d8 (0xf2b bytes) 000368[0] 1785395048.699967 read 0x60bff6 thru 0x60db67 (0x1b72 bytes) 000370[1] 1785395048.825113 mapwrite 0x52605a thru 0x5261f8 (0x19f bytes) 000371[1] 1785395049.042158 mapread 0x230ad6 thru 0x231f6d (0x1498 bytes) 000373[1] 1785395049.198951 read 0x28cad7 thru 0x28cd22 (0x24c bytes) 000376[0] 1785395049.278903 mapread 0x7faa98 thru 0x7fbc61 (0x11ca bytes) 000378[1] 1785395049.346861 mapwrite 0x963f56 thru 0x9645b3 (0x65e bytes) 000379[1] 1785395049.593286 read 0x0fcbd1 thru 0x0fdc48 (0x1078 bytes) 000382[0] 1785395049.722676 read 0x86eb75 thru 0x86ffe2 (0x146e bytes) 000384[1] 1785395049.840352 mapread 0x709484 thru 0x70a722 (0x129f bytes) 000386[1] 1785395049.911268 write 0x7c18fc thru 0x7c2aeb (0x11f0 bytes) 000387[0] 1785395049.935710 mapwrite 0x938ced thru 0x938d87 (0x9b bytes) 000388[0] 1785395050.214066 trunc from 0x9645b4 to 0x8c9e44 (0x9a76f bytes) 000389[0] 1785395050.365603 read 0x58a158 thru 0x58b9c9 (0x1872 bytes) 000390[0] 1785395050.414694 mapwrite 0x62e296 thru 0x62f1d3 (0xf3e bytes) 000392[0] 1785395050.540754 write 0x87b466 thru 0x87b6c4 (0x25f bytes) 000396[0] 1785395050.635752 read 0x2d0936 thru 0x2d1fc8 (0x1693 bytes) 000401[1] 1785395050.815224 trunc from 0x8c9e45 to 0x33740d (0x592a37 bytes) 000405[1] 1785395051.011133 read 0x09467c thru 0x095d0a (0x168f bytes) 000406[0] 1785395051.162805 read 0x024384 thru 0x0256a6 (0x1323 bytes) 000408[1] 1785395051.309589 trunc from 0x33740e to 0x78e9f1 (0x4575e4 bytes) 000409[1] 1785395051.416703 mapread 0x311e73 thru 0x313049 (0x11d7 bytes) 000410[1] 1785395051.473077 trunc from 0x78e9f2 to 0x2d7463 (0x4b758e bytes) 000411[1] 1785395051.507843 trunc from 0x2d7464 to 0x503d53 (0x22c8f0 bytes) 000412[1] 1785395051.545133 mapwrite 0x4b27be thru 0x4b3481 (0xcc4 bytes) 000414[1] 1785395051.645267 read 0x180b78 thru 0x181b8b (0x1014 bytes) 000415[1] 1785395051.654386 trunc from 0x503d54 to 0x0933db (0x470978 bytes) 000417[0] 1785395051.714244 read 0x03c07e thru 0x03d56b (0x14ee bytes) 000418[1] 1785395051.746553 read 0x01e7d4 thru 0x01fcba (0x14e7 bytes) 000421[1] 1785395051.794106 mapread 0x07a71c thru 0x07a7e1 (0xc6 bytes) 000422[1] 1785395051.795555 mapread 0x046d8c thru 0x046e79 (0xee bytes) 000423[0] 1785395051.796584 write 0x00d262 thru 0x00dab8 (0x857 bytes) 000424[0] 1785395051.822624 trunc from 0x0933dc to 0x23e1d0 (0x1aadf5 bytes) 000426[0] 1785395051.869537 write 0x879483 thru 0x87b43f (0x1fbd bytes) 000428[1] 1785395051.873961 write 0x9b42f9 thru 0x9b48b0 (0x5b8 bytes) 000430[0] 1785395051.908292 trunc from 0x9b48b1 to 0x2375c1 (0x77d2ef bytes) truncating to largest ever: 0x9f3fb4 000432[1] 1785395051.954938 trunc from 0x2375c2 to 0x9f3fb3 (0x7bc9f2 bytes) 000434[0] 1785395052.012158 mapread 0x38bafd thru 0x38cd06 (0x120a bytes) 000437[1] 1785395052.106845 trunc from 0x9f3fb4 to 0x98f405 (0x64bae bytes) 000438[0] 1785395052.169693 read 0x929461 thru 0x92ab8e (0x172e bytes) 000439[1] 1785395052.189674 write 0x90f424 thru 0x91031f (0xefc bytes) 000440[1] 1785395052.324150 trunc from 0x98f406 to 0x4e92d8 (0x4a612d bytes) 000443[0] 1785395052.470067 read 0x249b2a thru 0x24b4b4 (0x198b bytes) 000445[0] 1785395052.542709 write 0x5231aa thru 0x523fb6 (0xe0d bytes) 000447[1] 1785395052.590642 mapwrite 0x0315a9 thru 0x0328dc (0x1334 bytes) 000448[0] 1785395052.655017 trunc from 0x523fb7 to 0x2bb03e (0x268f78 bytes) 000450[1] 1785395052.703153 trunc from 0x2bb03f to 0x7fbe29 (0x540deb bytes) 000451[1] 1785395052.736969 write 0x1c83bf thru 0x1c9dc5 (0x1a07 bytes) 000452[1] 1785395052.764747 write 0x837fbc thru 0x83975b (0x17a0 bytes) 000453[1] 1785395052.782003 mapread 0x301751 thru 0x3036c8 (0x1f78 bytes) 000455[0] 1785395052.796432 read 0x0befc2 thru 0x0c003d (0x107c bytes) 000457[0] 1785395052.837532 trunc from 0x83975c to 0x479a91 (0x3bfcca bytes) 000459[0] 1785395052.902226 mapwrite 0x1b2718 thru 0x1b3fd4 (0x18bd bytes) 000461[1] 1785395052.983392 trunc from 0x479a92 to 0x15149f (0x3285f2 bytes) 000463[1] 1785395053.053818 write 0x975fcd thru 0x9775cd (0x1601 bytes) 000467[0] 1785395053.103987 mapread 0x60a012 thru 0x60a1cc (0x1bb bytes) 000468[1] 1785395053.145143 write 0x582103 thru 0x582740 (0x63e bytes) 000471[1] 1785395053.178364 trunc from 0x9775ce to 0x71abb8 (0x25ca15 bytes) 000472[0] 1785395053.209491 write 0x193555 thru 0x1937da (0x286 bytes) 000473[1] 1785395053.238698 mapread 0x6e2ee1 thru 0x6e4a7e (0x1b9e bytes) 000474[1] 1785395053.272318 read 0x55a5a0 thru 0x55a824 (0x285 bytes) 000476[0] 1785395053.299673 mapwrite 0x647878 thru 0x648383 (0xb0c bytes) 000477[1] 1785395053.387295 trunc from 0x71abb9 to 0x96286e (0x247cb6 bytes) 000480[1] 1785395053.457191 write 0x703eab thru 0x705db9 (0x1f0f bytes) 000487[1] 1785395053.540286 write 0x21ebed thru 0x21f2b0 (0x6c4 bytes) 000488[0] 1785395053.563418 write 0x6b6717 thru 0x6b83a1 (0x1c8b bytes) 000490[1] 1785395053.610441 trunc from 0x96286f to 0x5f540a (0x36d464 bytes) 000492[0] 1785395053.668861 mapwrite 0x5aaa7c thru 0x5ac7be (0x1d43 bytes) 000494[0] 1785395053.763611 mapwrite 0x6fbc0b thru 0x6fd6db (0x1ad1 bytes) 000495[1] 1785395054.021290 mapread 0x49cb6a thru 0x49e90e (0x1da5 bytes) 000497[1] 1785395054.178494 mapread 0x2b2d56 thru 0x2b313b (0x3e6 bytes) 000498[1] 1785395054.254586 mapread 0x28829e thru 0x2890ad (0xe10 bytes) 000499[1] 1785395054.295520 mapwrite 0x0a608f thru 0x0a63a5 (0x317 bytes) 000500[1] 1785395054.502189 read 0x658f59 thru 0x65964f (0x6f7 bytes) All operations completed A-OK! PASS 16b (42s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 03:04:22 (1785395062) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (6s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 03:04:28 (1785395068) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.24803 s, 6.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.12875 s, 7.4 MB/s ldlm.namespaces.MGC192.168.204.152@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9316c5236800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9316c79f6000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9316c79f6000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9316c79f6000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.0066 s, 5.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.86888 s, 17.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.11031 s, 25.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0821 s, 10.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.2455 s, 6.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.14481 s, 17.1 MB/s PASS 16d (83s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 03:05:51 (1785395151) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.501023 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165631 s, 6.3 MB/s PASS 16e (11s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 03:06:02 (1785395162) TIMEOUT PASS 16f (32s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 03:06:34 (1785395194) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 03:07:05 (1785395225) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (10s) == sanityn test 16i: read after truncate file ============ 03:07:15 (1785395235) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (12s) == sanityn test 16j: race dio with buffered i/o ========== 03:07:27 (1785395247) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.28923 s, 5.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.58395 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.34359 s, 4.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.03909 s, 12.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07448 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.6177 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.82862 s, 5.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.61676 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.25985 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.0034 s, 5.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.16482 s, 5.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.05161 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.33851 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.49418 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.06821 s, 5.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.039 s, 3.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.21246 s, 3.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.39043 s, 3.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.86186 s, 3.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (72s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 03:08:39 (1785395319) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Jul 30 03:09:58 EDT 2026 + fsx 2 done: rc=0 Thu Jul 30 03:09:58 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Thu Jul 30 03:09:59 EDT 2026 + fsx 4 done: rc=0 Thu Jul 30 03:09:59 EDT 2026 + fsx 5 done: rc=0 Thu Jul 30 03:09:59 EDT 2026 PASS 16k (96s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 03:10:15 (1785395415) fail_loc=0x8000030a PASS 17 (21s) == sanityn test 18: mmap sanity check =========================================================================================== 03:10:36 (1785395436) mmap test1: basic mmap operation (PASS, 0.14377s) mmap test2: MAP_PRIVATE not write back (PASS, 0.092687s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.1s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7501s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.347s) mmap test6: check mmap write/read content on two nodes (PASS, 0.5767s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.389s) mmap test11: multi-thread shared mmap (PASS, 1.891s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1698s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 03:11:13 (1785395473) oleg452-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 03:11:19 (1785395479) PASS 20 (10s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 03:11:29 (1785395489) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 03:11:38 (1785395498) now is 1785395502 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 new atime is 1785395564 PASS 23 (74s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 03:12:53 (1785395573) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (10s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 03:13:02 (1785395582) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 03:13:11 (1785395591) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 03:13:20 (1785395600) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 03:13:24 (1785395604) utime: good mknod times 1785395605 <= 1785395606 <= 1785395606 for /mnt/lustre/f26a utime: good mknod times 1785395605 <= 1785395606 <= 1785395606 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (10s) == sanityn test 26b: sync mtime between ost and mds ====== 03:13:35 (1785395615) PASS 26b (12s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 03:13:46 (1785395626) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 03:13:48.000000000 -0400 Birth: 2026-07-30 03:13:49.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 03:13:48.000000000 -0400 Birth: 2026-07-30 03:13:49.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 03:13:59 (1785395639) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 03:14:10 (1785395650) fail_loc=0x314 PASS 31a (11s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 03:14:21 (1785395661) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.152@tcp:/lustre 7542784 40960 7495680 1% /mnt/lustre2 PASS 31b (28s) == sanityn test 31r: open-rename(replace) race =========== 03:14:49 (1785395689) -rw-r--r-- 1 root root 0 Jul 30 03:14 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (13s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 03:15:02 (1785395702) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43203 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 03:15:12 (1785395712) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 03:15:22 (1785395722) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 03:15:28 (1785395728) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.12197 s, 24.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56262 s, 33.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.28225 s, 16.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.62717 s, 14.5 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06017 s, 25.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.52559 s, 20.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74713 s, 30.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.7557 s, 29.9 MB/s PASS 32c (41s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 03:16:10 (1785395770) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 03:16:16 (1785395776) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (5s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 03:16:21 (1785395781) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (5s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 03:16:26 (1785395786) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (5s) == sanityn test 34: no lock timeout under IO ============= 03:16:32 (1785395792) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9316c5236800.ost_server_uuid in IDLE state after 0 sec PASS 34 (183s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 03:19:35 (1785395975) fail_loc=0x80000317 Race attempt 0 Wait for 50519 50537 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50519 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.152@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.152@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 447 (time 1785395996.00 total 10.02 last 44.59) - open/close 863 (time 1785396006.03 total 20.05 last 41.50) - open/close 1263 (time 1785396016.04 total 30.06 last 39.95) - open/close 1665 (time 1785396026.04 total 40.07 last 40.18) - open/close 2128 (time 1785396036.05 total 50.07 last 46.29) - open/close 2640 (time 1785396046.05 total 60.07 last 51.17) - open/close 2952 (time 1785396056.06 total 70.09 last 31.16) - open/close 3407 (time 1785396066.07 total 80.09 last 45.47) - open/close 3855 (time 1785396076.08 total 90.10 last 44.76) total: 4000 open/close in 93.25 seconds: 42.89 ops/second PASS 35 (104s) == sanityn test 36: handle ESTALE/open-unlink correctly == 03:21:19 (1785396079) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.907548 s, 57.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7428096) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2365 s, 42.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7425024) after(7476224) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.913617 s, 57.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7476224) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.931697 s, 56.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12881 s, 46.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7424000) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05666 s, 49.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7428096) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13544 s, 46.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7436288) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.36661 s, 22.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15291 s, 45.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7418880) after(7470080) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59655 s, 32.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7470080) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78609 s, 29.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7424000) after(7496704) PASS 36 (420s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 03:28:19 (1785396499) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 - create 526 (time 1785396516.06 total 10.00 last 52.59) - create 1001 (time 1785396526.06 total 20.01 last 47.49) - create 1525 (time 1785396536.06 total 30.01 last 52.40) - create 2409 (time 1785396546.06 total 40.01 last 88.38) - create 3273 (time 1785396556.07 total 50.01 last 86.35) - create 4020 (time 1785396566.07 total 60.02 last 74.67) - create 4802 (time 1785396576.10 total 70.05 last 77.97) - create 5747 (time 1785396586.11 total 80.05 last 94.44) - create 6830 (time 1785396596.11 total 90.05 last 108.29) - create 7638 (time 1785396606.12 total 100.07 last 80.69) - create 8461 (time 1785396616.13 total 110.07 last 82.25) - create 9311 (time 1785396626.20 total 120.14 last 84.40) total: 10000 create in 124.93 seconds: 80.05 ops/second PASS 37 (141s) == sanityn test 39a: file mtime does not change after rename ========================================================== 03:30:40 (1785396640) repeat after cancel_lru_locks PASS 39a (11s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 03:30:52 (1785396652) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 03:31:03 (1785396663) repeat after cancel_lru_locks PASS 39c (10s) == sanityn test 39d: sync write should update mtime ====== 03:31:12 (1785396672) fail_loc=0x411 fail_loc=0 PASS 39d (10s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 03:31:24 (1785396684) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (20s) == sanityn test 40c: pdirops: link and others ======================================================================== 03:31:44 (1785396704) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 03:32:05 (1785396725) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (21s) == sanityn test 40e: pdirops: rename and others ======================================================================== 03:32:26 (1785396746) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (28s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 03:32:54 (1785396774) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (25s) == sanityn test 41b: pdirops: create vs create ======================================================================== 03:33:19 (1785396799) open: File exists Conflict PASS 41b (23s) == sanityn test 41c: pdirops: create vs link ======================================================================== 03:33:42 (1785396822) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (17s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 03:33:59 (1785396839) Conflict PASS 41d (20s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 03:34:20 (1785396860) Conflict PASS 41e (19s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 03:34:40 (1785396880) Conflict PASS 41f (17s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 03:34:57 (1785396897) Conflict PASS 41g (19s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 03:35:16 (1785396916) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (25s) == sanityn test 41i: reint_open: create vs create ======== 03:35:42 (1785396942) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1287s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:57:09 (1785398229) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (18s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:57:28 (1785398248) open: File exists Conflict PASS 42b (18s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:57:45 (1785398265) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (18s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:58:03 (1785398283) Conflict PASS 42d (19s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:58:23 (1785398303) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (25s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:58:48 (1785398328) Conflict PASS 42f (22s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:59:10 (1785398350) Conflict PASS 42g (18s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:59:28 (1785398368) Conflict PASS 42h (19s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:59:47 (1785398387) PASS 43a (86s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 04:01:14 (1785398474) Conflict PASS 43b (18s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 04:01:33 (1785398493) Conflict PASS 43c (25s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 04:01:57 (1785398517) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (18s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 04:02:15 (1785398535) Conflict PASS 43e (18s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 04:02:33 (1785398553) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (18s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 04:02:52 (1785398572) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (18s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 04:03:09 (1785398589) Conflict PASS 43h (18s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 04:03:27 (1785398607) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 04:03:32 (1785398612) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (168s) == sanityn test 43k: unlink vs create ==================== 04:06:20 (1785398780) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1670s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 04:34:11 (1785400451) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jul 30 04:34:22 EDT 2026 PASS 44a (18s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 04:34:29 (1785400469) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 04:34:47 (1785400487) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 04:35:03 (1785400503) Conflict PASS 44d (17s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 04:35:21 (1785400521) Conflict PASS 44e (18s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 04:35:40 (1785400540) Conflict PASS 44f (19s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 04:35:59 (1785400559) Conflict PASS 44g (19s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 04:36:18 (1785400578) Conflict PASS 44h (19s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 04:36:37 (1785400597) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 04:36:41 (1785400601) PASS 45a (168s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 04:39:29 (1785400769) Conflict PASS 45b (24s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 04:39:54 (1785400794) Conflict PASS 45c (25s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 04:40:18 (1785400818) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (25s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 04:40:43 (1785400843) Conflict PASS 45e (26s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 04:41:09 (1785400869) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (18s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 04:41:28 (1785400888) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (17s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 04:41:45 (1785400905) Conflict PASS 45h (17s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 04:42:03 (1785400923) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (5s) == sanityn test 45j: read vs rename ====================== 04:42:08 (1785400928) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1731s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 05:10:59 (1785402659) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (18s) == sanityn test 46b: pdirops: link vs create ======================================================================== 05:11:17 (1785402677) open: File exists Conflict PASS 46b (18s) == sanityn test 46c: pdirops: link vs link ======================================================================== 05:11:36 (1785402696) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (20s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 05:11:55 (1785402715) Conflict PASS 46d (23s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 05:12:19 (1785402739) Conflict PASS 46e (21s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 05:12:41 (1785402761) Conflict PASS 46f (18s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 05:12:59 (1785402779) Conflict PASS 46g (19s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 05:13:18 (1785402798) Conflict PASS 46h (21s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 05:13:39 (1785402819) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 05:13:44 (1785402824) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 05:13:49 (1785402829) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 05:13:54 (1785402834) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 05:13:58 (1785402838) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 05:14:03 (1785402843) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 05:14:09 (1785402849) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 05:14:14 (1785402854) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (7s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 05:14:21 (1785402861) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.125643 s, 81.5 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (17s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 05:14:39 (1785402879) 0+1 records in 0+1 records out 158 bytes copied, 0.0690166 s, 2.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (219146) - No such process PASS 51a (13s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 05:14:52 (1785402892) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0754689 s, 13.6 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258411 s, 4.1 MB/s 3145728 PASS 51b (25s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 05:15:18 (1785402918) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0242075 s, 42.3 kB/s PASS 51c (20s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 05:15:37 (1785402937) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096292 s, 10.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (15s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 05:15:52 (1785402952) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (15s) == sanityn test 54: rename locking ======================= 05:16:07 (1785402967) llite.lustre-ffff9316c5236800.xattr_cache=1 llite.lustre-ffff9316c79f6000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:10.000000000 -0400 Modify: 2026-07-30 05:16:10.000000000 -0400 Change: 2026-07-30 05:16:10.000000000 -0400 Birth: 2026-07-30 05:16:11.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:18.000000000 -0400 Modify: 2026-07-30 05:16:19.000000000 -0400 Change: 2026-07-30 05:16:19.000000000 -0400 Birth: 2026-07-30 05:16:18.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:25.000000000 -0400 Modify: 2026-07-30 05:16:25.000000000 -0400 Change: 2026-07-30 05:16:25.000000000 -0400 Birth: 2026-07-30 05:16:25.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:25.000000000 -0400 Modify: 2026-07-30 05:16:26.000000000 -0400 Change: 2026-07-30 05:16:26.000000000 -0400 Birth: 2026-07-30 05:16:25.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:32.000000000 -0400 Modify: 2026-07-30 05:16:33.000000000 -0400 Change: 2026-07-30 05:16:33.000000000 -0400 Birth: 2026-07-30 05:16:32.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:16:32.000000000 -0400 Modify: 2026-07-30 05:16:33.000000000 -0400 Change: 2026-07-30 05:16:33.000000000 -0400 Birth: 2026-07-30 05:16:32.000000000 -0400 PASS 54 (39s) == sanityn test 55a: rename vs unlink target dir ========= 05:16:46 (1785403006) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (20s) == sanityn test 55b: rename vs unlink source dir ========= 05:17:06 (1785403026) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (22s) == sanityn test 55c: rename vs unlink orphan target dir == 05:17:29 (1785403049) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 05:17:49 (1785403069) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (19s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 05:18:09 (1785403089) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (7s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 05:18:15 (1785403095) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (25s) == sanityn test 55g: rename: race with trylock =========== 05:18:41 (1785403121) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (30s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 05:19:11 (1785403151) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1785403154 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.391276 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (168s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 05:21:59 (1785403319) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1785403326 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.79269 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (174s) == sanityn test 60: Verify data_version behaviour ======== 05:24:53 (1785403493) PASS 60 (15s) == sanityn test 70a: cd directory && rm directory ======== 05:25:09 (1785403509) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 05:25:19 (1785403519) PASS 70b (13s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 05:25:33 (1785403533) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0233751 s, 2.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 05:25:38 (1785403538) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0264252 s, 2.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (6s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 05:25:44 (1785403544) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (5s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 05:25:49 (1785403549) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0180892 s, 3.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 05:25:54 (1785403554) llite.lustre-ffff9316c5236800.xattr_cache=1 llite.lustre-ffff9316c79f6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 05:26:03 (1785403563) llite.lustre-ffff9316c5236800.xattr_cache=1 llite.lustre-ffff9316c79f6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (11s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 05:26:14 (1785403574) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0454563 s, 22.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00591609 s, 173 kB/s 231996: get lock2 231998: get lock1 231998: done 231998: get lock2 231996: done 231996: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231998: cannot get lock2: Resource deadlock avoided 231998: exit rc=0 231996: done 231996: exit rc=0 PASS 74 (16s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 05:26:30 (1785403590) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0869071 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398507 s, 26.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232579 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (18s) == sanityn test 76: Verify MDT open_files listing ======== 05:26:48 (1785403608) opening files: ........ closing files: ........ PASS 76 (98s) == sanityn test 77a: check FIFO NRS policy =============== 05:28:26 (1785403706) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.49557 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076584 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677253 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101998 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348626 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544681 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228792 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267297 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282063 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378687 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184565 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373497 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404173 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25393 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313737 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313848 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405816 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671024 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105818 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108094 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974376 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160594 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100763 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874894 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12556 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121853 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152723 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129034 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151417 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151745 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179939 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21105 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183813 s, 5.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (17s) == sanityn test 77b: check CRR-N NRS policy ============== 05:28:43 (1785403723) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436492 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766381 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280815 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888325 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449693 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970312 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151712 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272028 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552439 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461167 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907819 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277504 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116065 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393592 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222784 s, 4.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413623 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427123 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09867 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111374 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108702 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123976 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956874 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113338 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107927 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101705 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154279 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963658 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17743 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115619 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109159 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135752 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963387 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141448 s, 7.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.440819 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280265 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100707 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121665 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104344 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943663 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214526 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177896 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562558 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456992 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126344 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268556 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331561 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181754 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183197 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284316 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263608 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102251 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772973 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798113 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895892 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786487 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093786 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152129 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930242 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164564 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113475 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817557 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947238 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990458 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160969 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104506 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130507 s, 8.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (19s) == sanityn test 77c: check ORR NRS policy ================ 05:29:02 (1785403742) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.481993 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641432 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585706 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16164 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185647 s, 5.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720849 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102561 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149734 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112857 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1601 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057842 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333258 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296831 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165536 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06624 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055783 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134007 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136782 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136574 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148975 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978919 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166549 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155332 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628525 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133025 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124766 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978463 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108761 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185095 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106788 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910247 s, 115 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14351 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165516 s, 6.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.586528 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521966 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132156 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145738 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222805 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32994 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393058 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390704 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425163 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353401 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27337 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272976 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299791 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449804 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377374 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398393 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567595 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126528 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105123 s, 99.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129118 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115919 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177902 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305917 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253979 s, 413 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158981 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172178 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225193 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154741 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123624 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946979 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532372 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298633 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098404 s, 10.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.485877 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823738 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091811 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21494 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174546 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197215 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249791 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268861 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21596 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285297 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189286 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219541 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177963 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237193 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246182 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35089 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385375 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458928 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572727 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572346 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117057 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599653 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621374 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787728 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881627 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836902 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116295 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948036 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103674 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804453 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810856 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896778 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120356 s, 8.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (24s) == sanityn test 77d: check TRR nrs policy ================ 05:29:26 (1785403766) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.365263 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693521 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485965 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804857 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18806 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206015 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115571 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190243 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222659 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184127 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256938 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297722 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2423 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308709 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313451 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249715 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362077 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113175 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545214 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00202751 s, 517 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00201763 s, 520 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00209095 s, 501 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226904 s, 462 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262512 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269822 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165285 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144442 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127445 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11746 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101895 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112541 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766326 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312208 s, 33.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265657 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574732 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879583 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105816 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196019 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504693 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219386 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194641 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231777 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217369 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34582 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274519 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264054 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348424 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443647 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433628 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.599732 s, 1.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153704 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15896 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227194 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160059 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146828 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267495 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27922 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137354 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167325 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274401 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205372 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181646 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215111 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198199 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197628 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321649 s, 3.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410393 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676805 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619217 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116717 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239954 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242434 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186093 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191803 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280278 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242198 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255527 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344431 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30592 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354642 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38925 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390664 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380691 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264882 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116406 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809216 s, 130 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486166 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144817 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983702 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078321 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515923 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144557 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150812 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103406 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318153 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798166 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159962 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268787 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100324 s, 105 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (20s) == sanityn test 77e: check TBF NID nrs policy ============ 05:29:46 (1785403786) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.52@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3587 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295377 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139215 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523259 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019959 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486142 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734353 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204269 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203373 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170856 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234424 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120517 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247969 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172026 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173946 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216569 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29289 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222791 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126459 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11634 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738985 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209677 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274083 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291542 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113008 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264273 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290526 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290375 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153386 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289755 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320432 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289971 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317599 s, 3.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.592188 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158703 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025447 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171177 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944289 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276623 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308078 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355954 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395172 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35875 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207588 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438097 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396123 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265467 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268476 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288049 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.539508 s, 1.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328283 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150307 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999543 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110605 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13689 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681035 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100576 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145534 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118772 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171882 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900666 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12512 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184933 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133102 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130782 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216861 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=5 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.378834 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730368 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125138 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211676 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338204 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256064 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318283 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227718 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247009 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33857 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28226 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241597 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402476 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450347 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354603 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41056 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.601231 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177438 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359458 s, 29.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249106 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252331 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208957 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230252 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110877 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931245 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858743 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117196 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922339 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787874 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695149 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507682 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072479 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887578 s, 11.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.624473 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733578 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123308 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611688 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253873 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964456 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296097 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1594 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790382 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302928 s, 3.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252553 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254037 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148587 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279113 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515798 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3429 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.552682 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209616 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329064 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173741 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196274 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125374 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113793 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118534 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791446 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288595 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220489 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336025 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14992 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253205 s, 414 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.002541 s, 413 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0099115 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176553 s, 59.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (34s) == sanityn test 77f: check TBF JobID nrs policy ========== 05:30:20 (1785403820) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482917 s, 34.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761999 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101385 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166982 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239141 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190805 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157433 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817226 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321334 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318303 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238696 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343011 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126934 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399084 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304896 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239129 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29272 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051251 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120129 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813796 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935721 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725411 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136126 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105475 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274059 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297259 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175121 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192982 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137487 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236524 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211135 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537442 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313571 s, 33.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447888 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960159 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797427 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831612 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106758 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170664 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039573 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207348 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300792 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360279 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258278 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300567 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348766 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113019 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37473 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495339 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153287 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00983333 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130985 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181247 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00277752 s, 378 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00866263 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101508 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129983 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124772 s, 84.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112694 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197492 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25492 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861229 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202346 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266427 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213531 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258631 s, 40.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.215661 s, 77.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227203 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227022 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120916 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303647 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463153 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494873 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530856 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340382 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976116 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113548 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076469 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151775 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881726 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805355 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119743 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127172 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347364 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344962 s, 304 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461103 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631767 s, 16.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957307 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563593 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502401 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302965 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540663 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537105 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649212 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759731 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766659 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539501 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669339 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068396 s, 15.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.364958 s, 46.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369352 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419413 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620742 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137766 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151178 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249923 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231491 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319209 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558386 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107259 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324569 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104468 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256391 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343314 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487831 s, 21.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145111 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944632 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03116 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352638 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090552 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601558 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981591 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1064 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120832 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125389 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112134 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817072 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922188 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817889 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941893 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959754 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103677 s, 10.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (34s) == sanityn test 77g: Change TBF type directly ============ 05:30:54 (1785403854) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 05:31:09 (1785403869) oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 PASS 77h (17s) == sanityn test 77i: Change rank of TBF rule ============= 05:31:27 (1785403887) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (30s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 05:31:56 (1785403916) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.636885 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655836 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441545 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646719 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132793 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135592 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847942 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558258 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233752 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186879 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144653 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286693 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289027 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301285 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253427 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386845 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307893 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754537 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979046 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898359 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786774 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607883 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08202 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762549 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805351 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925155 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161806 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884215 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189778 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121606 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128209 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132326 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238769 s, 4.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.2146 s, 6.5 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6958 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (75s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 05:33:12 (1785403992) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30222 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802722 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102456 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598182 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148812 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167653 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165887 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169158 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153311 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154112 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160977 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22151 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215047 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250083 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186108 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28301 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26858 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888183 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115267 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116708 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938254 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754324 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832565 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118794 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146412 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094303 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100072 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156344 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148902 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10354 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105215 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127814 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143263 s, 7.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.9805 s, 5.0 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8504 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33051 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367273 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232301 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147488 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174849 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314359 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246128 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323211 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266712 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429529 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382353 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.519326 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430813 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428892 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45253 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.566234 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.624062 s, 1.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196034 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16957 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17903 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213464 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220476 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213413 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2383 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227362 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260196 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280252 s, 3.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243447 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243957 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236842 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272777 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351345 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334259 s, 3.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.9149 s, 5.0 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2257 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (174s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 05:36:06 (1785404166) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30895 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862013 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109047 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116737 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095021 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274155 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199687 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14048 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141785 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359514 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304683 s, 3.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322642 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233415 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431977 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326971 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321681 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482771 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478501 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979378 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103437 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828095 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114817 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118954 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861947 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775713 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621754 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946791 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100349 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14034 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104721 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119758 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100313 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13317 s, 7.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.1074 s, 5.0 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8684 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31102 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191732 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25999 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292369 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258888 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163604 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349952 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344008 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189254 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128633 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294727 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239297 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31896 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265438 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.530107 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290354 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394772 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157015 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114581 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165221 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131212 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173081 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133992 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19545 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145347 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185371 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235388 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13984 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136935 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166347 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172323 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198356 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231355 s, 4.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 22.0552 s, 4.8 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.004 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (173s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 05:38:59 (1785404339) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.423338 s, 39.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313884 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350969 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494715 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076918 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290886 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834595 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149962 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212938 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14619 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236642 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111106 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188409 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761092 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108421 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176935 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183544 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110958 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819726 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113556 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112723 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11068 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946773 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115891 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808098 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175859 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125691 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131496 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120424 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251018 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126201 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144103 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145814 s, 7.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0156 s, 8.7 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3255 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.52@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.52@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.600896 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907279 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998961 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579496 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105112 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228963 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374718 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36399 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385922 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345898 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36741 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396653 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407073 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345119 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481518 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480101 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473413 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184782 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623613 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104049 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580268 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227251 s, 461 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151279 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173532 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00760146 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125412 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179534 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264698 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14279 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170187 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159742 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280076 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169861 s, 6.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.089 s, 6.5 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4932 s, 10.0 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.52@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.977673 s, 17.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161642 s, 64.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411795 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565991 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758391 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2044 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201441 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227526 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228661 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272456 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265713 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300062 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32345 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334482 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292997 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306519 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30168 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450358 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632237 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499739 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515889 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685626 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539732 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619162 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613236 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728384 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948632 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09311 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970688 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10229 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14184 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103201 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101855 s, 10.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.8189 s, 6.6 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.5007 s, 16.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.575954 s, 29.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491067 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942723 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112379 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214617 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116648 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198139 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135564 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228105 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128324 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261773 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194917 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254159 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262211 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196769 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311785 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230909 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108562 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692993 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696265 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685623 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740686 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160233 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905754 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171408 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944555 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996443 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15929 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094784 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113691 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126955 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217187 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138388 s, 7.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.0651 s, 6.5 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2497 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.3254 s, 7.3 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2837 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.605932 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308589 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462642 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123125 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12577 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087442 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273132 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30813 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386801 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37551 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433099 s, 2.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438945 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312642 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446505 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333951 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468971 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430112 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959827 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.008973 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837573 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122646 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119034 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147426 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721136 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777009 s, 135 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351369 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151177 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010595 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218391 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195825 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043723 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00212896 s, 493 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105796 s, 99.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4991 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7089 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.6754 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11156 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100995 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792136 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231052 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195869 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131359 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223874 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312156 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164455 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333068 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292289 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22119 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356253 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349027 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381294 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464596 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135737 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893816 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131232 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156381 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130279 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132965 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142386 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113026 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165493 s, 6.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142696 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237699 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262622 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267651 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196341 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263138 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303461 s, 3.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.3728 s, 6.4 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1656 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.935812 s, 17.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848494 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989328 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122698 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279024 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24075 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186221 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210811 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365062 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263222 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292664 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370735 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391507 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374636 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289191 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294908 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491095 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645064 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702624 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592488 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863333 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746072 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811209 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845061 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115472 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112035 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166976 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145234 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209192 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149634 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158141 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22806 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195109 s, 5.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.9404 s, 6.2 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6755 s, 9.8 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.7812 s, 6.6 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6249 s, 9.9 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (534s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 05:47:53 (1785404873) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01243 s, 16.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451584 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519662 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920835 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098395 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431702 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119002 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601413 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697924 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386305 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187263 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148093 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17702 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21431 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265934 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263759 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238679 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947961 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990228 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532124 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981571 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587686 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621913 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104672 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068019 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672348 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706936 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726705 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782691 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827126 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808846 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939399 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101173 s, 10.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.144 s, 6.5 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95963 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (64s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 05:48:57 (1785404937) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.52@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.52@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.745006 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257221 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620279 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296508 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106061 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123686 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186388 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141253 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174079 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204973 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168236 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26205 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282064 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283175 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308538 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323941 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292238 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134572 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181235 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178953 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137946 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160847 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145215 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176179 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160812 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203876 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174734 s, 6.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203717 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207171 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211518 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211045 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264686 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22554 s, 4.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.8856 s, 6.2 MB/s Write runtime is 18 s, speed is 5.555555 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6869 s, 9.8 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (70s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 05:50:07 (1785405007) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.52@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.668872 s, 25.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453411 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774173 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12737 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113229 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190976 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171345 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232201 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206968 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276825 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295481 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236782 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302096 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336387 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240192 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247843 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398413 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c5236800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302866 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836501 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141624 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104623 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103337 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256857 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229405 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085693 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924487 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102725 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103309 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244629 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127997 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149262 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005329 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332506 s, 31.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.873 s, 6.6 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.29235 s, 16.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (63s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 05:51:10 (1785405070) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.730494 s, 23.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709174 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620923 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875822 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342172 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334157 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440127 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616733 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144092 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222653 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22871 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698325 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052788 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143142 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202498 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199071 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337364 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104141 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813258 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135912 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785741 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138551 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109461 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993427 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915166 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100194 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161304 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103859 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163679 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151742 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128004 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150356 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216062 s, 4.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.5058 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.32 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 18.8281 s, 5.6 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3075 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (120s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 05:53:10 (1785405190) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21662 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636917 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726781 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994938 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199806 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226508 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323165 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174055 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268881 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23785 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160051 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421882 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299764 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433077 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311574 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267958 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340474 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432783 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020506 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252762 s, 415 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120579 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224497 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389551 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278077 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856628 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197296 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303292 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286221 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353247 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179575 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661205 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700656 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118436 s, 8.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.738 s, 4.8 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0755 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (90s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 05:54:40 (1785405280) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.98506 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134081 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217818 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25431 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289166 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349684 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932781 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211606 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334451 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351954 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.545816 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228674 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126373 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270198 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201396 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215701 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20097 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9316c5236800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658546 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661681 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431686 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239105 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204741 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120619 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277082 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159423 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160533 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229785 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179796 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197863 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210789 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22122 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221541 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241806 s, 4.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.9056 s, 7.5 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7718 s, 9.7 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg452-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.16728 s, 14.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286224 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800456 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101124 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159987 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161654 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131168 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164551 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173974 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193548 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180224 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173759 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169734 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204837 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173602 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207295 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243756 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322777 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388855 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413863 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103635 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589013 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528044 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788252 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840293 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858463 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922599 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15493 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985382 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114549 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120124 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126203 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134387 s, 7.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.9983 s, 5.8 MB/s Write runtime is 19 s, speed is 5.263157 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3339 s, 10.1 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.1514 s, 5.5 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1339 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (191s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 05:57:52 (1785405472) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg452-client.virtnet /mnt/lustre (opts:) Stopping client oleg452-client.virtnet /mnt/lustre opts: Stopping clients: oleg452-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg452-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg452-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg452-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.52,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg452-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.52,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg452-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.52,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/66909/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.52,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg452-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre Starting client oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre Started clients oleg452-client.virtnet: 192.168.204.152@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Started clients oleg452-client.virtnet: 192.168.204.152@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9316c8c7f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9316d1c83000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316c8c7f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316d1c83000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.880696 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595948 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619699 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399164 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545188 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186482 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273571 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319797 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325402 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348273 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297928 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301818 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375425 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315166 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421783 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391052 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381413 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c8c7f000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9316c8c7f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345124 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738788 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519855 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590022 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925037 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640898 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503746 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107224 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109674 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111104 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713276 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15778 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117035 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118285 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958517 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100791 s, 10.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7188 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.15498 s, 17.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.936128 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749694 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655673 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103106 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232525 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246649 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103041 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356672 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154115 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429091 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838657 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173799 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407315 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206318 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09916 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186336 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693468 s, 15.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c8c7f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9316c8c7f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184785 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188598 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162888 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165703 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148989 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150538 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216159 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145077 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165548 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16932 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173437 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184357 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185481 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190667 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195664 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207738 s, 5.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0906 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4884 s, 10.0 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (295s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 06:02:47 (1785405767) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (24s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 06:03:11 (1785405791) On MGS 192.168.204.152, active = nodemap.active=1 On MGS 192.168.204.152, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15566 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105451 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951573 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149754 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112556 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111925 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242003 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233843 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541383 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35229 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221982 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187709 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133811 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312759 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10232 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603535 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122129 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330743 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126057 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111091 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125307 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533311 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054489 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638388 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447765 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681406 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741638 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609268 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939662 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958378 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105855 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108545 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128414 s, 8.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.9494 s, 6.6 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62872 s, 15.8 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.152, TBF.id = On MGS 192.168.204.152, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05618 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503845 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129048 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178737 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261575 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319831 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35941 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230753 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349905 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296819 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472636 s, 2.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362451 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441638 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515719 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488692 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c8c7f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9316c8c7f000.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52718 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.664749 s, 1.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c8c7f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833103 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630324 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076182 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116721 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562473 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570844 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080173 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103614 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12628 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974233 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773754 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098998 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153412 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217644 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203671 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14484 s, 7.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9869 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.4149 s, 43.4 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg452-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 On MGS 192.168.204.152, active = nodemap.active=1 On MGS 192.168.204.152, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0055 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445261 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950526 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105547 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25678 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268893 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273427 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367127 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38131 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429349 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485054 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453177 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459121 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.528079 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52807 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.552811 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.624875 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9316c8c7f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9316c8c7f000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109308 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168763 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633584 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155423 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811152 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177193 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736664 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108525 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104038 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105529 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111633 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120959 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120987 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115643 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119034 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276417 s, 3.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.3176 s, 7.9 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1752 s, 10.3 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.152, TBF.id = On MGS 192.168.204.152, active = nodemap.active=0 PASS 77kj (246s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:07:17 (1785406037) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (16s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:07:33 (1785406053) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg452-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.04116 s, 259 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg452-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0668 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (63s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:08:36 (1785406116) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5085 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108157 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141489 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266449 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294665 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325896 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280708 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423904 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361276 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417791 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385569 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471591 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472819 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.578722 s, 1.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480677 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.61369 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.57147 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417444 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110774 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139431 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124966 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150025 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128372 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149827 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792575 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014752 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112018 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128533 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311893 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111835 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181053 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153059 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310989 s, 33.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg452-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg452-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.9459 s, 5.8 MB/s Write runtime is 19 s, speed is 5.263157 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.57539 s, 15.9 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.437561 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05991 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895396 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101023 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139284 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149805 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132867 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072635 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491223 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129913 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114355 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135275 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153636 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797225 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145811 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284963 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133179 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646328 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676974 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647239 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823306 s, 12.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043382 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817908 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907066 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759962 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942628 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124279 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064642 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759288 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635974 s, 165 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104218 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859265 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250516 s, 41.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg452-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg452-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.961 s, 7.0 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.33409 s, 16.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (125s) == sanityn test 77o: Changing rank should not panic ====== 06:10:41 (1785406241) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (21s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:11:02 (1785406262) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (180s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:14:03 (1785406443) mds.MDS.mdt.nrs_policies=tbf oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg452-client: oleg452-server: ssh exited with exit code 36 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 oleg452-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg452-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (51s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 06:14:54 (1785406494) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273173 kill md_thread xargs PASS 77r (67s) == sanityn test 77s: Check TBF LRU shrinker ============== 06:16:01 (1785406561) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (104s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:17:45 (1785406665) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (14s) == sanityn test 79: xattr: intent error ================== 06:17:59 (1785406679) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935587 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 06:18:02.000000000 -0400 Modify: 2026-07-30 06:18:02.000000000 -0400 Change: 2026-07-30 06:18:02.000000000 -0400 Birth: 2026-07-30 06:18:03.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (23s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:18:22 (1785406702) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (5s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:18:28 (1785406708) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (5s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:18:33 (1785406713) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:18:38 (1785406718) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (5s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:18:43 (1785406723) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (7s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 06:18:50 (1785406730) - open/close 360 (time 1785406751.55 total 10.01 last 35.97) - open/close 1074 (time 1785406761.55 total 20.02 last 71.34) - open/close 1549 (time 1785406771.56 total 30.03 last 47.45) - open/close 2135 (time 1785406781.57 total 40.03 last 58.58) total: 2500 open/close in 43.96 seconds: 56.87 ops/second snapshot_time 1785406909.075299128 secs.nsecs start_time 1785406737.011292179 secs.nsecs elapsed_time 172.064006949 secs.nsecs open 2505 samples [usecs] 149 20763 3432107 9995949959 close 2504 samples [usecs] 159 9437 804115 806006995 mknod 2501 samples [usecs] 537 17867 2994207 7672632449 unlink 2500 samples [usecs] 706 141576 6912533 170930632725 mkdir 2 samples [usecs] 1033 1440 2473 3140689 rmdir 2 samples [usecs] 8160 13007 21167 235767649 rename 2500 samples [usecs] 442 78451 22323037 451805519685 getattr 3764 samples [usecs] 9 39781 389803 3546733823 parallel_rename_file 2500 samples [usecs] 442 78451 22323037 451805519685 crossdir_rename 2500 samples [usecs] 442 78451 22323037 451805519685 PASS 81d (185s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 06:21:55 (1785406915) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 PASS 82 (10s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 06:22:05 (1785406925) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 06:22:10 (1785406930) fail_loc=0x8000060b PASS 84 (16s) == sanityn test 85: Lustre API root cache race =========== 06:22:27 (1785406947) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1785406949 Finishing test test1 at 1785406956 PASS 85 (15s) == sanityn test 90: open/create and unlink striped directory ========================================================== 06:22:42 (1785406962) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 06:22:46 (1785406966) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (5s) == sanityn test 92: create remote directory under orphan directory ========================================================== 06:22:51 (1785406971) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 06:22:56 (1785406976) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5721 0x1659 0x240000400 1 5683 0x1633 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5722 0x165a 0x240000400 1 5684 0x1634 0x280000400 rc1=0 and rc2=0 PASS 93 (23s) == sanityn test 94: signal vs CP callback race =========== 06:23:18 (1785406998) osc.lustre-OST0000-osc-ffff9316c8c7f000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9316d1c83000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9316c8c7f000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9316d1c83000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.374572 s, 21.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.9167 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289536 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0797766 s, 51.3 kB/s osc.lustre-OST0000-osc-ffff9316c8c7f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9316d1c83000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316c8c7f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9316d1c83000.idle_timeout=debug PASS 94 (20s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 06:23:38 (1785407018) llite.lustre-ffff9316c8c7f000.fast_read=0 llite.lustre-ffff9316d1c83000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154943 s, 13.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9316c8c7f000.fast_read=1 llite.lustre-ffff9316d1c83000.fast_read=1 PASS 95a (27s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 06:24:06 (1785407046) llite.lustre-ffff9316c8c7f000.fast_read=0 llite.lustre-ffff9316d1c83000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00661765 s, 1.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9316c8c7f000.fast_read=1 llite.lustre-ffff9316d1c83000.fast_read=1 PASS 95b (17s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 06:24:23 (1785407063) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 06:24:28 (1785407068) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203447 s, 201 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (12s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 06:24:41 (1785407081) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.438949 s, 4.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (19s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 06:25:01 (1785407101) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.356792 s, 5.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (16s) == sanityn test 100e: DoM: read on open and file size ==== 06:25:16 (1785407116) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0110867 s, 1.1 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 06:25:26 (1785407126) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0328751 s, 125 kB/s PASS 101a (11s) == sanityn test 101b: Discard DoM data on rename ========= 06:25:36 (1785407136) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187611 s, 218 kB/s PASS 101b (10s) == sanityn test 101c: Discard DoM data on close-unlink === 06:25:47 (1785407147) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0319283 s, 128 kB/s PASS 101c (16s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 06:26:03 (1785407163) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 04 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf04:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (17s) == sanityn test 103: Test size correctness with lockahead ========================================================== 06:26:19 (1785407179) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1785407183 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1785407187 Finishing test test23 at 1785407188 PASS 103 (20s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 06:26:39 (1785407199) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (6s) == sanityn test 105: Glimpse and lock cancel race ======== 06:26:45 (1785407205) fail_loc=0x416 PASS 105 (27s) == sanityn test 106a: Verify the btime via statx() ======= 06:27:12 (1785407232) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (5s) == sanityn test 106b: Glimpse RPCs test for statx ======== 06:27:17 (1785407237) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0492248 s, 20.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938123 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 06:27:19.000000000 -0400 Modify: 2026-07-30 06:27:19.000000000 -0400 Change: 2026-07-30 06:27:19.000000000 -0400 Birth: 2026-07-30 06:27:19.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938123 -rw-r--r-- 1 0 0 1785407239 1785407239 1785407239 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938123 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 06:27:19.000000000 -0400 Modify: 2026-07-30 06:27:19.000000000 -0400 Change: 2026-07-30 06:27:19.000000000 -0400 Birth: 2026-07-30 06:27:19.000000000 -0400 1785407239 1024 1 PASS 106b (10s) == sanityn test 106c: Verify statx attributes mask ======= 06:27:27 (1785407247) PASS 106c (13s) == sanityn test 107a: Basic grouplock conflict =========== 06:27:40 (1785407260) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.586948 s, 17.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5731 0x1663 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 First grouplock blocks second one PASS 107a (17s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 06:27:57 (1785407277) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.61012 s, 17.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5691 0x163b 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (19s) == sanityn test 108a: lseek: parallel updates ============ 06:28:16 (1785407296) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.020462 s, 489 kB/s llite.lustre-ffff9316c8c7f000.hybrid_io=0 llite.lustre-ffff9316d1c83000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 5.1326 s, 1.6 MB/s PASS 108a (17s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 06:28:33 (1785407313) 192.168.204.152@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 192.168.204.152@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 PASS 109 (70s) == sanityn test 110: do not grant another lock on resend ========================================================== 06:29:44 (1785407384) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 06:29:46.000000000 -0400 Modify: 2026-07-30 06:29:46.000000000 -0400 Change: 2026-07-30 06:29:46.000000000 -0400 Birth: 2026-07-30 06:29:46.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg452-client.virtnet oleg452-client.virtnet PASS 110 (85s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 06:31:09 (1785407469) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 06:31:13 (1785407473) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 06:31:17 (1785407477) lustre-OST0000-osc-ffff9316edcbc000 active. lustre-OST0001-osc-ffff9316edcbc000 active. lustre-OST0000-osc-ffff9316edcbc000 active. lustre-OST0001-osc-ffff9316edcbc000 active. MGC192.168.204.152@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 06:31:27 (1785407487) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 06:31:31 (1785407491) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 06:31:36 (1785407496) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 06:31:41 (1785407501) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.071887 s, 285 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.151514 s, 135 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 06:31:53 (1785407513) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2318 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.259 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2029 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3624 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2836 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3265 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3042 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1848 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.307 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3429 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3849 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3678 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3503 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4014 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3615 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4236 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6037 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5283 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9316c79f3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9316c79f3800.ost_server_uuid in IDLE state after 0 sec PASS 200 (89s) cleanup: ====================================================== == sanityn test complete, duration 13226 sec ============= 06:33:24 (1785407604) === sanityn: start cleanup 06:33:27 (1785407607) === Stopping clients: oleg452-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg452-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 06:39:38 (1785407978) ===