-----============= acceptance-small: sanityn ============----- Thu Jul 30 01:54:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg232-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 01:54:28 (1785390868) === Starting client oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Mount client oleg232-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Started clients oleg232-client.virtnet: 192.168.202.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg232-client.virtnet: executing check_config_client /mnt/lustre oleg232-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg232-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff965984c46000.idle_timeout=debug osc.lustre-OST0000-osc-ffff965991e9f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff965984c46000.idle_timeout=debug osc.lustre-OST0001-osc-ffff965991e9f800.idle_timeout=debug disable quota as required oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:55:13 (1785390913) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580457 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7613] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:55:16 (1785390916) PASS 0 (11s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:55:27 (1785390927) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:55:36 (1785390936) -rw-r--r-- 1 root root 0 Jul 30 01:55 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (9s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:55:45 (1785390945) -rw-r--r-- 1 root root 0 Jul 30 01:55 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:55:53 (1785390953) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:56:02 (1785390962) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (11s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:56:12 (1785390972) total 1 -rwxrwxrwx 1 root root 0 Jul 30 01:55 f2a -rwxrwxrwx 1 root root 0 Jul 30 01:55 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 30 01:55 f2a -rwxrwxrwx 1 root root 0 Jul 30 01:55 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0151714 s, 33.7 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:56:20 (1785390980) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (5s) == sanityn test 2g: check blocks update on sync write ==== 01:56:25 (1785390985) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.561581 s, 3.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:56:34 (1785390994) PASS 3 (10s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:56:44 (1785391004) PASS 4 (12s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:56:56 (1785391016) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (10s) == sanityn test 6: remove of open file on other node ============================================================================ 01:57:06 (1785391026) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:57:14 (1785391034) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (9s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:57:23 (1785391043) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (10s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:57:33 (1785391053) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 01:57:43 (1785391063) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948699 s, 11.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0640897 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:57:52 (1785391072) 1+0 records in 1+0 records out 1 byte copied, 0.00616481 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0209138 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0278357 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.03895 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0266302 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0210744 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0187272 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102087 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118468 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0245602 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.022007 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00997211 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:58:02 (1785391082) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0523636 s, 58.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0588297 s, 69.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000254178 s, 12.1 MB/s PASS 10b (10s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:58:12 (1785391092) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:58:21 (1785391101) warning: '-runtime' deprecated, use '-t runtime' instead - link 231 (time 1785391114.17 total 10.01 last 23.08) - link 473 (time 1785391124.24 total 20.07 last 24.04) - link 797 (time 1785391134.26 total 30.10 last 32.32) - link 1134 (time 1785391144.27 total 40.11 last 33.68) - link 1451 (time 1785391154.30 total 50.14 last 31.59) - link 1767 (time 1785391164.35 total 60.19 last 31.45) - link 1988 (time 1785391174.37 total 70.20 last 22.07) - link 2198 (time 1785391184.40 total 80.24 last 20.93) - link 2437 (time 1785391194.45 total 90.29 last 23.78) total: 2651 link in 100.02 seconds: 26.50 ops/second using seed 3163239260 running for 100 seconds - stat 2966 (time 1785391116 ; total 11 ; last 11) - stat 7471 (time 1785391127 ; total 22 ; last 11) - stat 10000 (time 1785391131 ; total 26 ; last 4) - stat 15126 (time 1785391142 ; total 37 ; last 11) - stat 20000 (time 1785391152 ; total 47 ; last 10) - stat 23234 (time 1785391163 ; total 58 ; last 11) - stat 25235 (time 1785391174 ; total 69 ; last 11) - stat 27031 (time 1785391185 ; total 80 ; last 11) - stat 28837 (time 1785391196 ; total 91 ; last 11) - stat 30000 (time 1785391203 ; total 98 ; last 7) total: 30345 stats in 100 seconds: 303.450012 stats/second - unlinked 0 (time 1785391155 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2651) error: No such file or directory total: 2650 unlinks in 79 seconds: 33.544304 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21422) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21424) - No such process PASS 12 (139s) == sanityn test 13: test directory page revocation ======= 02:00:41 (1785391241) f13 f13 PASS 13 (12s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:00:53 (1785391253) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 22610 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (16s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:01:09 (1785391269) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 23204 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (15s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:01:25 (1785391285) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23784 Terminated $DIR1/$tdir/sleep 60 PASS 14b (10s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:01:34 (1785391294) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24366 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:01:43 (1785391303) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24950 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:01:54 (1785391314) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (69s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 02:03:03 (1785391383) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.10231 s, 7.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.625577 s, 13.4 MB/s Chance of close/open is 1 in 50 Seed set to 1390 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x901fbf 000003[0] 1785391391.330317 trunc from 00000000 to 0x901fbe (0x901fbf bytes) 000004[1] 1785391391.438253 trunc from 0x901fbf to 0x06b23a (0x896d84 bytes) 000008[0] 1785391391.541377 write 0x334ced thru 0x33e075 (0x9389 bytes) 000013[1] 1785391391.673813 read 0x2b7cab thru 0x2bc026 (0x437c bytes) 000014[0] 1785391391.738930 read 0x0a3eab thru 0x0a7bae (0x3d04 bytes) 000016[1] 1785391391.763711 read 0x176611 thru 0x17eb0c (0x84fc bytes) 000017[1] 1785391391.844038 mapwrite 0x790eac thru 0x79cc2e (0xbd83 bytes) 000022[0] 1785391392.090090 trunc from 0x79cc2f to 0x6a0398 (0xfc896 bytes) 000023[0] 1785391392.135657 trunc from 0x6a0399 to 0x6507fe (0x4fb9a bytes) 000025[1] 1785391392.167788 write 0x8d3a7e thru 0x8e0d96 (0xd319 bytes) 000026[1] 1785391392.190093 mapwrite 0x9f000a thru 0x9f73f1 (0x73e8 bytes) 000029[0] 1785391392.306891 write 0x863fde thru 0x869013 (0x5036 bytes) 000031[0] 1785391392.345431 mapread 0x066d2e thru 0x06cfce (0x62a1 bytes) 000033[0] 1785391392.391944 mapwrite 0x4cdb90 thru 0x4d5a51 (0x7ec2 bytes) 000034[0] 1785391392.461629 mapwrite 0x60e87d thru 0x617fe8 (0x976c bytes) 000041[0] 1785391392.574841 mapread 0x51548c thru 0x51f739 (0xa2ae bytes) 000044[0] 1785391392.627111 mapread 0x6a48a8 thru 0x6a9c1f (0x5378 bytes) 000045[0] 1785391392.660552 mapread 0x6c028e thru 0x6cd846 (0xd5b9 bytes) 000046[1] 1785391392.682637 read 0x56d6b0 thru 0x57b89e (0xe1ef bytes) 000048[0] 1785391392.720909 trunc from 0x9f73f2 to 0x745850 (0x2b1ba1 bytes) 000049[0] 1785391392.790109 read 0x6212d4 thru 0x625af8 (0x4825 bytes) 000050[0] 1785391392.801419 trunc from 0x745851 to 0x54c5e8 (0x1f9268 bytes) 000051[0] 1785391392.818272 mapwrite 0x7a1c4c thru 0x7a1e32 (0x1e7 bytes) 000053[0] 1785391392.896046 mapwrite 0x6a4d5c thru 0x6a8546 (0x37eb bytes) 000057[1] 1785391392.979016 mapwrite 0x529955 thru 0x530acd (0x7179 bytes) 000059[0] 1785391393.065800 read 0x14d609 thru 0x14e847 (0x123f bytes) 000062[0] 1785391393.107603 write 0x143bfe thru 0x151af1 (0xdef4 bytes) 000065[0] 1785391393.151690 mapwrite 0x1d6da3 thru 0x1d9435 (0x2693 bytes) truncating to largest ever: 0x9a163b 000068[1] 1785391393.293548 trunc from 0x7a1e33 to 0x9a163a (0x1ff808 bytes) 000069[0] 1785391393.335054 trunc from 0x9a163b to 0x84b279 (0x1563c1 bytes) 000070[0] 1785391393.376841 read 0x716b91 thru 0x71df8c (0x73fc bytes) 000071[0] 1785391393.387400 mapread 0x582a23 thru 0x58e8fd (0xbedb bytes) 000072[0] 1785391393.425934 trunc from 0x84b27a to 0x72c076 (0x11f203 bytes) 000073[0] 1785391393.447599 mapread 0x060dc9 thru 0x06e4cd (0xd705 bytes) 000075[0] 1785391393.492074 trunc from 0x72c077 to 0x3d7c6b (0x35440b bytes) 000076[0] 1785391393.523835 write 0x7cb7f2 thru 0x7d8a37 (0xd246 bytes) 000077[1] 1785391393.548595 read 0x2b27bf thru 0x2b515e (0x29a0 bytes) 000080[1] 1785391393.604721 mapread 0x2efa54 thru 0x2f6cb0 (0x725d bytes) 000081[1] 1785391393.630304 mapwrite 0x599902 thru 0x59dcd5 (0x43d4 bytes) 000086[0] 1785391393.785099 trunc from 0x7d8a38 to 0x1449fb (0x69403c bytes) 000087[0] 1785391393.872741 mapwrite 0x8ae8bf thru 0x8be1b3 (0xf8f5 bytes) 000090[1] 1785391394.084618 read 0x541178 thru 0x54bbd5 (0xaa5e bytes) 000092[1] 1785391394.134722 mapread 0x234509 thru 0x2433a0 (0xee98 bytes) 000093[0] 1785391394.179549 mapread 0x4a2668 thru 0x4a9ab2 (0x744b bytes) 000096[0] 1785391394.211277 mapread 0x383032 thru 0x390fe8 (0xdfb7 bytes) 000098[1] 1785391394.235953 trunc from 0x8be1b4 to 0x474f87 (0x44922c bytes) 000099[1] 1785391394.318764 read 0x3a4b63 thru 0x3a6d16 (0x21b4 bytes) 000103[1] 1785391394.359209 mapwrite 0x126014 thru 0x12b93b (0x5928 bytes) 000105[1] 1785391394.437453 trunc from 0x474f88 to 0x8dc70c (0x467785 bytes) 000106[1] 1785391394.480937 write 0x0504d8 thru 0x057c81 (0x77aa bytes) 000110[1] 1785391394.524490 mapread 0x69f28a thru 0x6a0cd3 (0x1a4a bytes) 000114[1] 1785391394.588066 mapwrite 0x1b2985 thru 0x1b6a19 (0x4095 bytes) 000117[1] 1785391394.748280 write 0x467857 thru 0x46e309 (0x6ab3 bytes) 000118[0] 1785391394.802271 trunc from 0x8dc70d to 0x525237 (0x3b74d5 bytes) 000120[1] 1785391394.874969 mapwrite 0x425226 thru 0x427d1c (0x2af7 bytes) 000122[0] 1785391394.990899 mapread 0x0a02e5 thru 0x0a49f0 (0x470c bytes) 000123[0] 1785391395.040224 read 0x4b761a thru 0x4c1160 (0x9b47 bytes) 000124[0] 1785391395.120089 write 0x3c81ca thru 0x3d24f3 (0xa32a bytes) 000125[1] 1785391395.121290 read 0x49660e thru 0x4a5d03 (0xf6f6 bytes) 000126[0] 1785391395.137970 mapread 0x386a83 thru 0x38aaa0 (0x401e bytes) 000128[1] 1785391395.169459 read 0x20b21f thru 0x20dbb2 (0x2994 bytes) 000129[1] 1785391395.219725 trunc from 0x525238 to 0x372971 (0x1b28c6 bytes) 000132[0] 1785391395.430569 write 0x622951 thru 0x62d072 (0xa722 bytes) 000134[1] 1785391395.523691 mapwrite 0x4629c7 thru 0x472889 (0xfec3 bytes) 000135[1] 1785391395.806601 mapread 0x413f4c thru 0x417335 (0x33ea bytes) 000136[0] 1785391395.838880 read 0x476cac thru 0x478a3c (0x1d91 bytes) 000141[1] 1785391396.016861 read 0x5ac092 thru 0x5b794c (0xb8bb bytes) 000148[1] 1785391396.152268 write 0x5b5289 thru 0x5bbc34 (0x69ac bytes) 000151[1] 1785391396.182452 write 0x0e63ba thru 0x0ebf76 (0x5bbd bytes) 000152[1] 1785391396.250400 write 0x8b8123 thru 0x8b92db (0x11b9 bytes) 000153[0] 1785391396.256623 trunc from 0x8b92dc to 0x217a50 (0x6a188b bytes) 000154[1] 1785391396.419635 trunc from 0x217a51 to 0x7dfa3f (0x5c7fef bytes) 000156[1] 1785391396.564765 write 0x73dc82 thru 0x7419fa (0x3d79 bytes) 000158[0] 1785391396.674520 read 0x25a4cd thru 0x25aa12 (0x546 bytes) 000162[0] 1785391396.905429 trunc from 0x7dfa40 to 0x99108d (0x1b164e bytes) 000163[1] 1785391397.133081 trunc from 0x99108e to 0x4bc2b1 (0x4d4ddc bytes) 000164[0] 1785391397.237089 mapwrite 0x08a8bd thru 0x08f4e4 (0x4c28 bytes) 000167[1] 1785391397.647876 mapread 0x262655 thru 0x267a22 (0x53ce bytes) 000174[1] 1785391398.114500 mapwrite 0x2a9bdb thru 0x2b932b (0xf751 bytes) 000175[0] 1785391398.383324 trunc from 0x4bc2b2 to 0x5a186b (0xe55ba bytes) 000180[0] 1785391398.527824 mapread 0x4022cd thru 0x4089ea (0x671e bytes) 000181[0] 1785391398.546473 mapwrite 0x2f8be9 thru 0x2fbd47 (0x315f bytes) 000184[0] 1785391398.647990 read 0x43bc3a thru 0x449db7 (0xe17e bytes) 000186[1] 1785391398.707426 mapread 0x489145 thru 0x48c3fc (0x32b8 bytes) 000187[0] 1785391398.784123 mapwrite 0x094410 thru 0x09c14f (0x7d40 bytes) 000188[1] 1785391398.951300 read 0x09aba4 thru 0x09b883 (0xce0 bytes) 000189[1] 1785391399.022652 mapwrite 0x49d47c thru 0x4a8f40 (0xbac5 bytes) 000191[1] 1785391399.247573 mapwrite 0x5e2e1b thru 0x5e922f (0x6415 bytes) 000193[1] 1785391399.543740 trunc from 0x5e9230 to 0x241b4c (0x3a76e3 bytes) 000195[1] 1785391399.634506 trunc from 0x241b4d to 0x87de64 (0x63c318 bytes) 000197[0] 1785391399.770927 mapwrite 0x992f30 thru 0x9a02dd (0xd3ae bytes) 000200[1] 1785391400.094353 write 0x864e64 thru 0x867b80 (0x2d1d bytes) 000204[1] 1785391400.301730 trunc from 0x9a02de to 0x27ddf9 (0x7224e4 bytes) 000205[0] 1785391400.394345 mapread 0x040dbd thru 0x047641 (0x6885 bytes) 000207[0] 1785391400.495358 write 0x348007 thru 0x35395d (0xb957 bytes) 000210[1] 1785391400.651305 trunc from 0x35395e to 0x220050 (0x13390d bytes) 000211[1] 1785391400.807846 trunc from 0x220051 to 0x47a119 (0x25a0c9 bytes) 000212[0] 1785391400.868645 mapread 0x2aa00b thru 0x2b7b2f (0xdb25 bytes) 000219[1] 1785391401.048932 mapread 0x34b367 thru 0x359af6 (0xe790 bytes) 000220[0] 1785391401.193603 read 0x265e17 thru 0x269c9c (0x3e86 bytes) 000221[1] 1785391401.206781 mapread 0x335585 thru 0x33e09b (0x8b17 bytes) 000223[0] 1785391401.290321 read 0x3a2bed thru 0x3ab0bf (0x84d3 bytes) 000225[1] 1785391401.314154 trunc from 0x47a11a to 0x6949bb (0x21a8a2 bytes) 000227[1] 1785391401.471668 read 0x5a0d3a thru 0x5af55c (0xe823 bytes) 000231[0] 1785391401.626445 mapwrite 0x3e6b50 thru 0x3edff2 (0x74a3 bytes) 000233[1] 1785391402.006695 mapread 0x2fedd3 thru 0x30b070 (0xc29e bytes) 000234[1] 1785391402.142458 mapwrite 0x0d56d2 thru 0x0e13fa (0xbd29 bytes) 000235[0] 1785391402.472932 mapwrite 0x337e12 thru 0x34715b (0xf34a bytes) 000238[0] 1785391402.876637 trunc from 0x6949bc to 0x354f81 (0x33fa3a bytes) 000239[1] 1785391403.104014 write 0x7fe4d1 thru 0x805c42 (0x7772 bytes) 000240[1] 1785391403.172573 write 0x36537f thru 0x366393 (0x1015 bytes) 000241[0] 1785391403.218571 mapwrite 0x70a771 thru 0x7115ba (0x6e4a bytes) 000244[0] 1785391403.715118 trunc from 0x805c43 to 0x480992 (0x3852b0 bytes) 000247[1] 1785391403.844552 read 0x1d94fd thru 0x1dd0ca (0x3bce bytes) 000249[0] 1785391403.941744 read 0x4776a8 thru 0x480992 (0x92eb bytes) 000251[1] 1785391403.956022 read 0x09e806 thru 0x0a2efa (0x46f5 bytes) 000254[0] 1785391403.992802 mapwrite 0x209dd2 thru 0x219cb9 (0xfee8 bytes) 000255[0] 1785391404.398426 mapwrite 0x49e960 thru 0x4a2c1f (0x42c0 bytes) 000260[0] 1785391404.766725 mapwrite 0x0a84cd thru 0x0b0086 (0x7bba bytes) 000263[1] 1785391405.110726 mapwrite 0x15c0e3 thru 0x15fe63 (0x3d81 bytes) 000265[1] 1785391405.524576 trunc from 0x4a2c20 to 0x53a4d2 (0x978b3 bytes) 000267[1] 1785391405.678076 trunc from 0x53a4d3 to 0x99289e (0x4583cc bytes) 000271[1] 1785391405.850541 read 0x7b6ceb thru 0x7c28ab (0xbbc1 bytes) 000272[0] 1785391405.968348 mapwrite 0x53e0ca thru 0x54a8b4 (0xc7eb bytes) 000273[0] 1785391406.253692 trunc from 0x99289f to 0x661a82 (0x330e1c bytes) 000278[0] 1785391406.557727 mapwrite 0x228a42 thru 0x237538 (0xeaf7 bytes) 000281[0] 1785391406.928140 read 0x4e2c5b thru 0x4e2e5e (0x204 bytes) 000282[1] 1785391406.975326 write 0x39bb92 thru 0x39c96e (0xddd bytes) 000284[1] 1785391407.060052 mapwrite 0x5250b2 thru 0x534ab8 (0xfa07 bytes) 000285[1] 1785391407.316140 trunc from 0x661a83 to 0x1241e7 (0x53d89b bytes) 000286[1] 1785391407.364858 mapread 0x055c18 thru 0x05886c (0x2c55 bytes) 000288[1] 1785391407.401828 trunc from 0x1241e8 to 0x74aa25 (0x62683e bytes) 000291[1] 1785391407.511146 mapread 0x4f172d thru 0x501384 (0xfc58 bytes) 000292[0] 1785391407.560836 write 0x3802e1 thru 0x38f67a (0xf39a bytes) 000293[0] 1785391407.622273 mapwrite 0x8eebbd thru 0x8f0168 (0x15ac bytes) 000297[0] 1785391408.064267 mapread 0x17aac3 thru 0x18198a (0x6ec8 bytes) 000299[1] 1785391408.206319 write 0x767806 thru 0x773407 (0xbc02 bytes) 000301[0] 1785391408.333721 mapread 0x4071e6 thru 0x413d10 (0xcb2b bytes) 000306[0] 1785391408.672412 mapread 0x5d0056 thru 0x5df9d8 (0xf983 bytes) 000307[0] 1785391408.815133 mapread 0x394cb7 thru 0x39dad5 (0x8e1f bytes) 000309[1] 1785391408.916656 read 0x7a49c4 thru 0x7a60a0 (0x16dd bytes) 000312[1] 1785391408.987628 mapwrite 0x5bf391 thru 0x5ca523 (0xb193 bytes) 000313[1] 1785391409.124135 read 0x10ff2d thru 0x11005b (0x12f bytes) 000315[1] 1785391409.187748 read 0x2d606d thru 0x2d8c50 (0x2be4 bytes) 000316[1] 1785391409.197343 trunc from 0x8f0169 to 0x6c0ddb (0x22f38d bytes) 000317[0] 1785391409.280499 mapwrite 0x4456b0 thru 0x4506f9 (0xb04a bytes) 000318[1] 1785391409.591396 mapread 0x5fa90a thru 0x603b88 (0x927f bytes) 000319[1] 1785391409.826393 read 0x6a1fca thru 0x6a9249 (0x7280 bytes) 000321[0] 1785391409.914416 write 0x92dd2b thru 0x931461 (0x3737 bytes) 000324[1] 1785391410.036987 mapwrite 0x7bcb44 thru 0x7bef9a (0x2457 bytes) 000325[0] 1785391410.436366 write 0x9db314 thru 0x9e289f (0x758c bytes) 000326[0] 1785391410.479351 trunc from 0x9e28a0 to 0x3ebe16 (0x5f6a89 bytes) 000327[1] 1785391410.580589 write 0x4d4c94 thru 0x4dd176 (0x84e3 bytes) 000329[0] 1785391410.661225 read 0x34d08c thru 0x356b2f (0x9aa4 bytes) 000330[1] 1785391410.869307 mapwrite 0x2485fe thru 0x254ba3 (0xc5a6 bytes) 000331[1] 1785391411.292570 mapread 0x1ae7f2 thru 0x1bba5a (0xd269 bytes) 000333[0] 1785391411.392764 read 0x34e027 thru 0x353534 (0x550e bytes) 000335[0] 1785391411.471730 mapread 0x09f529 thru 0x0a52e6 (0x5dbe bytes) 000337[1] 1785391411.550527 trunc from 0x4dd177 to 0x178e97 (0x3642df bytes) 000338[0] 1785391411.614838 mapread 0x083d68 thru 0x08d082 (0x931b bytes) 000341[1] 1785391411.789110 trunc from 0x178e98 to 0x43e211 (0x2c537a bytes) truncating to largest ever: 0x9b6403 000343[0] 1785391412.044474 trunc from 0x43e212 to 0x9b6402 (0x5781f1 bytes) 000344[1] 1785391412.308132 write 0x913569 thru 0x91955e (0x5ff6 bytes) 000346[0] 1785391412.486706 mapwrite 0x8574ba thru 0x862477 (0xafbe bytes) 000347[1] 1785391412.887612 mapread 0x90e3a3 thru 0x917f98 (0x9bf6 bytes) 000348[1] 1785391413.057465 mapread 0x157e63 thru 0x15fbf2 (0x7d90 bytes) 000350[1] 1785391413.126954 mapread 0x449823 thru 0x4571af (0xd98d bytes) 000351[1] 1785391413.269771 write 0x7eba73 thru 0x7f3d80 (0x830e bytes) 000355[1] 1785391413.467304 read 0x70ae23 thru 0x714f1e (0xa0fc bytes) 000357[0] 1785391413.528733 mapread 0x723939 thru 0x72cdf2 (0x94ba bytes) 000364[0] 1785391413.706446 mapread 0x3e848a thru 0x3f0222 (0x7d99 bytes) 000365[1] 1785391413.761247 mapwrite 0x2030c0 thru 0x20dccc (0xac0d bytes) 000368[0] 1785391414.109724 mapwrite 0x03801c thru 0x04095a (0x893f bytes) 000370[1] 1785391414.646756 mapread 0x1f4a27 thru 0x1f61a7 (0x1781 bytes) 000371[0] 1785391414.778942 mapwrite 0x1baf4f thru 0x1c9df5 (0xeea7 bytes) 000372[1] 1785391415.184784 mapread 0x1ad9bc thru 0x1b6886 (0x8ecb bytes) 000373[0] 1785391415.234722 mapwrite 0x696a9b thru 0x699766 (0x2ccc bytes) 000375[0] 1785391415.424897 mapread 0x42bbfe thru 0x43443a (0x883d bytes) 000379[0] 1785391415.523638 write 0x1088a4 thru 0x1121c3 (0x9920 bytes) 000385[0] 1785391415.795835 mapwrite 0x053918 thru 0x05a839 (0x6f22 bytes) 000386[1] 1785391416.163331 mapwrite 0x4d870b thru 0x4e6688 (0xdf7e bytes) 000387[0] 1785391416.508702 mapwrite 0x96f6a4 thru 0x976c2a (0x7587 bytes) 000389[1] 1785391416.811766 trunc from 0x9b6403 to 0x0e756d (0x8cee95 bytes) 000391[1] 1785391416.890366 read 0x04ffd1 thru 0x05bc1d (0xbc4d bytes) 000392[1] 1785391416.950528 mapwrite 0x18b882 thru 0x198a73 (0xd1f2 bytes) 000393[1] 1785391417.200656 read 0x06e076 thru 0x06e1ba (0x145 bytes) 000396[1] 1785391417.260194 mapwrite 0x13e74d thru 0x14bf1f (0xd7d3 bytes) 000397[0] 1785391417.393690 write 0x34dc97 thru 0x355398 (0x7702 bytes) 000399[0] 1785391417.466337 write 0x18cab4 thru 0x19231e (0x586b bytes) 000403[0] 1785391417.627056 mapread 0x018c08 thru 0x0247cb (0xbbc4 bytes) 000407[1] 1785391417.905059 trunc from 0x355399 to 0x3e6cd6 (0x9193e bytes) 000408[0] 1785391417.991836 write 0x7ad9c6 thru 0x7b28de (0x4f19 bytes) 000409[0] 1785391418.042515 write 0x838b43 thru 0x84237f (0x983d bytes) 000410[0] 1785391418.099680 mapread 0x2ee818 thru 0x2fb7a7 (0xcf90 bytes) 000411[0] 1785391418.239431 trunc from 0x842380 to 0x701bae (0x1407d1 bytes) 000419[0] 1785391418.355498 read 0x091921 thru 0x0969ac (0x508c bytes) 000420[0] 1785391418.452832 write 0x238087 thru 0x247822 (0xf79c bytes) 000422[0] 1785391418.580004 write 0x28b969 thru 0x29a462 (0xeafa bytes) 000423[1] 1785391418.630819 read 0x6e794e thru 0x6f338e (0xba41 bytes) 000424[0] 1785391418.695559 mapwrite 0x715c4d thru 0x7194b7 (0x386b bytes) 000425[0] 1785391418.961864 read 0x04621d thru 0x049293 (0x3077 bytes) 000426[1] 1785391418.973403 read 0x100e80 thru 0x10c73b (0xb8bc bytes) 000428[1] 1785391419.114436 trunc from 0x7194b8 to 0x781d4c (0x68895 bytes) 000430[1] 1785391419.240839 mapwrite 0x8ae05d thru 0x8b1fb2 (0x3f56 bytes) 000433[0] 1785391419.717156 read 0x2a9e29 thru 0x2b0890 (0x6a68 bytes) 000437[0] 1785391419.837505 read 0x100842 thru 0x10fdd1 (0xf590 bytes) 000438[1] 1785391419.868592 read 0x49ecef thru 0x49ef8d (0x29f bytes) 000440[1] 1785391419.917502 mapread 0x6b2d73 thru 0x6bfc04 (0xce92 bytes) 000441[0] 1785391420.002091 mapread 0x653ee1 thru 0x65a7b2 (0x68d2 bytes) 000446[1] 1785391420.099707 mapread 0x48ddd3 thru 0x496585 (0x87b3 bytes) 000447[0] 1785391420.185267 mapwrite 0x0a76bb thru 0x0a97b5 (0x20fb bytes) 000452[1] 1785391420.521227 mapread 0x0bc579 thru 0x0c4e99 (0x8921 bytes) 000453[0] 1785391420.641966 mapread 0x3367fb thru 0x340f2a (0xa730 bytes) 000454[1] 1785391420.727198 mapwrite 0x2117d1 thru 0x2138d5 (0x2105 bytes) 000455[0] 1785391420.817527 read 0x46b6ab thru 0x4752a8 (0x9bfe bytes) 000459[1] 1785391420.844146 mapread 0x76ed70 thru 0x77e747 (0xf9d8 bytes) 000461[0] 1785391420.886371 mapread 0x4f3cc4 thru 0x4fda1c (0x9d59 bytes) 000462[1] 1785391420.945164 write 0x05b1dc thru 0x066ef8 (0xbd1d bytes) 000468[1] 1785391421.111326 trunc from 0x8b1fb3 to 0x850c4b (0x61367 bytes) 000469[1] 1785391421.216869 read 0x2d7a38 thru 0x2d8ab2 (0x107b bytes) 000472[0] 1785391421.311741 read 0x4cca23 thru 0x4d53aa (0x8988 bytes) 000475[1] 1785391421.371161 write 0x5d67be thru 0x5df7a6 (0x8fe9 bytes) 000476[1] 1785391421.483851 mapread 0x0527a3 thru 0x05dec7 (0xb725 bytes) 000479[0] 1785391421.586361 mapwrite 0x385afd thru 0x386244 (0x748 bytes) 000482[0] 1785391421.878613 mapwrite 0x99d983 thru 0x9a48c3 (0x6f41 bytes) 000484[0] 1785391421.986971 trunc from 0x9a48c4 to 0x6a5553 (0x2ff370 bytes) 000485[0] 1785391422.029370 read 0x049e0c thru 0x0503c2 (0x65b7 bytes) 000486[0] 1785391422.042759 write 0x4e74d3 thru 0x4f0b46 (0x9674 bytes) 000487[0] 1785391422.055206 write 0x3f574d thru 0x3f8214 (0x2ac8 bytes) 000490[0] 1785391422.140269 mapread 0x31eefa thru 0x324aa1 (0x5ba8 bytes) 000496[0] 1785391422.190544 mapread 0x131733 thru 0x1335ed (0x1ebb bytes) 000497[0] 1785391422.214067 write 0x388428 thru 0x3948e9 (0xc4c2 bytes) 000498[0] 1785391422.235975 mapread 0x05a008 thru 0x06565f (0xb658 bytes) 000500[0] 1785391422.251419 read 0x559f6b thru 0x55c19c (0x2232 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1422 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1785391423.010950 write 0x0e5000 thru 0x0f4a25 (0xfa26 bytes) 000002[0] 1785391423.029644 mapread 0x0ae000 thru 0x0bde9d (0xfe9e bytes) 000005[1] 1785391423.091565 mapread 0x031000 thru 0x037367 (0x6368 bytes) 000014[1] 1785391423.155205 mapwrite 0x435000 thru 0x435473 (0x474 bytes) 000015[0] 1785391423.222355 mapread 0x2f3000 thru 0x2fccb7 (0x9cb8 bytes) 000018[0] 1785391423.258064 read 0x340000 thru 0x340d64 (0xd65 bytes) 000019[1] 1785391423.262235 read 0x30f000 thru 0x3109e6 (0x19e7 bytes) 000020[0] 1785391423.267574 mapwrite 0x6be000 thru 0x6c53ef (0x73f0 bytes) truncating to largest ever: 0x8478d9 000021[1] 1785391423.381364 trunc from 0x6c53f0 to 0x8478d8 (0x1824e9 bytes) 000022[0] 1785391423.427981 mapwrite 0x8c5000 thru 0x8c68cc (0x18cd bytes) 000024[0] 1785391423.668462 read 0x080000 thru 0x08ad8c (0xad8d bytes) 000025[0] 1785391423.710110 write 0x052000 thru 0x0524c0 (0x4c1 bytes) 000026[0] 1785391423.757439 mapwrite 0x3f8000 thru 0x404982 (0xc983 bytes) 000027[0] 1785391423.960023 write 0x934000 thru 0x936558 (0x2559 bytes) 000029[1] 1785391423.963571 write 0x550000 thru 0x555a7e (0x5a7f bytes) 000030[1] 1785391424.018391 write 0x41c000 thru 0x42a775 (0xe776 bytes) 000032[1] 1785391424.027060 mapwrite 0x6cd000 thru 0x6dbf99 (0xef9a bytes) 000035[0] 1785391424.289663 mapread 0x6b7000 thru 0x6b92c0 (0x22c1 bytes) 000037[0] 1785391424.394555 trunc from 0x936559 to 0x5ec6f1 (0x349e67 bytes) 000039[1] 1785391424.529857 write 0x99c000 thru 0x9a11a9 (0x51aa bytes) 000043[1] 1785391424.587545 trunc from 0x9a11aa to 0x67a891 (0x326918 bytes) 000044[0] 1785391424.656072 read 0x390000 thru 0x3916d6 (0x16d7 bytes) 000046[0] 1785391424.683637 trunc from 0x67a892 to 0x434a7a (0x245e17 bytes) 000047[1] 1785391424.763403 mapwrite 0x909000 thru 0x90ab4b (0x1b4c bytes) truncating to largest ever: 0x917168 000049[1] 1785391425.001991 trunc from 0x90ab4c to 0x917167 (0xc61c bytes) 000050[0] 1785391425.067124 write 0x70b000 thru 0x70b16f (0x170 bytes) 000051[1] 1785391425.104722 write 0x6c1000 thru 0x6cb1d7 (0xa1d8 bytes) 000053[0] 1785391425.140042 read 0x035000 thru 0x035d09 (0xd0a bytes) 000055[1] 1785391425.209435 mapread 0x56f000 thru 0x57e624 (0xf625 bytes) 000056[0] 1785391425.315716 mapread 0x030000 thru 0x03917b (0x917c bytes) 000058[1] 1785391425.354815 mapwrite 0x7c1000 thru 0x7c2b00 (0x1b01 bytes) 000062[0] 1785391425.563814 write 0x1e6000 thru 0x1e945a (0x345b bytes) 000063[0] 1785391425.607560 read 0x20a000 thru 0x20ab98 (0xb99 bytes) 000066[1] 1785391425.702563 trunc from 0x917168 to 0x787e99 (0x18f2ce bytes) 000067[0] 1785391425.760207 trunc from 0x787e9a to 0x118a88 (0x66f411 bytes) 000069[0] 1785391425.848979 mapwrite 0x1e5000 thru 0x1e89cb (0x39cc bytes) 000070[1] 1785391426.070056 mapread 0x1b3000 thru 0x1ba64a (0x764b bytes) 000072[0] 1785391426.116822 read 0x00d000 thru 0x013735 (0x6736 bytes) 000078[0] 1785391426.147070 trunc from 0x1e89cc to 0x6571c4 (0x46e7f9 bytes) 000082[0] 1785391426.211835 trunc from 0x6571c5 to 0x44045e (0x216d66 bytes) 000084[0] 1785391426.250504 mapread 0x17e000 thru 0x18c096 (0xe097 bytes) 000087[1] 1785391426.527959 read 0x339000 thru 0x343022 (0xa023 bytes) 000089[0] 1785391426.549981 trunc from 0x44045f to 0x3081f8 (0x138266 bytes) 000090[0] 1785391426.625304 mapread 0x1ba000 thru 0x1c5453 (0xb454 bytes) 000091[0] 1785391426.756720 mapwrite 0x5c3000 thru 0x5c6887 (0x3888 bytes) 000092[0] 1785391427.063912 read 0x27d000 thru 0x27e67c (0x167d bytes) 000095[1] 1785391427.137155 write 0x0e2000 thru 0x0f12a4 (0xf2a5 bytes) 000096[0] 1785391427.243841 write 0x750000 thru 0x75c586 (0xc587 bytes) 000097[1] 1785391427.297253 write 0x4f1000 thru 0x4f4b31 (0x3b32 bytes) 000100[0] 1785391427.359821 mapread 0x21b000 thru 0x225af6 (0xaaf7 bytes) 000102[1] 1785391427.508990 read 0x214000 thru 0x216609 (0x260a bytes) 000103[1] 1785391427.515225 write 0x569000 thru 0x571277 (0x8278 bytes) 000104[0] 1785391427.556513 trunc from 0x75c587 to 0x0b5b97 (0x6a69ef bytes) 000105[0] 1785391427.638767 trunc from 0x0b5b98 to 0x041837 (0x74360 bytes) 000106[0] 1785391427.669433 mapwrite 0x5ca000 thru 0x5cb4d1 (0x14d2 bytes) 000110[1] 1785391427.869983 write 0x3aa000 thru 0x3b1144 (0x7145 bytes) 000112[1] 1785391427.942118 mapread 0x0b3000 thru 0x0c0983 (0xd984 bytes) 000114[0] 1785391428.048104 write 0x6d4000 thru 0x6dd0ce (0x90cf bytes) 000117[0] 1785391428.082904 mapwrite 0x7cb000 thru 0x7ce050 (0x3051 bytes) 000118[0] 1785391428.429511 read 0x6ea000 thru 0x6f7546 (0xd547 bytes) 000119[0] 1785391428.455116 write 0x8a6000 thru 0x8a96e7 (0x36e8 bytes) 000122[0] 1785391428.504025 read 0x1ad000 thru 0x1ae776 (0x1777 bytes) 000123[0] 1785391428.529561 trunc from 0x8a96e8 to 0x5feee5 (0x2aa802 bytes) 000125[0] 1785391428.583273 mapwrite 0x576000 thru 0x57bac9 (0x5aca bytes) 000126[0] 1785391428.768415 mapread 0x4e0000 thru 0x4e179f (0x17a0 bytes) 000133[0] 1785391428.841396 mapwrite 0x597000 thru 0x59faca (0x8acb bytes) 000134[1] 1785391428.976790 trunc from 0x5feee6 to 0x794527 (0x195642 bytes) 000136[1] 1785391429.084280 trunc from 0x794528 to 0x6038ee (0x190c39 bytes) 000139[0] 1785391429.185230 mapwrite 0x8d7000 thru 0x8e3a21 (0xca22 bytes) 000144[0] 1785391429.517689 write 0x947000 thru 0x951bf7 (0xabf8 bytes) 000145[0] 1785391429.520115 write 0x422000 thru 0x4227cc (0x7cd bytes) 000146[0] 1785391429.549381 mapwrite 0x57d000 thru 0x583ae2 (0x6ae3 bytes) 000148[1] 1785391429.786887 mapread 0x847000 thru 0x84fc15 (0x8c16 bytes) 000149[1] 1785391429.871067 trunc from 0x951bf8 to 0x1544fa (0x7fd6fd bytes) 000151[1] 1785391429.907013 mapread 0x054000 thru 0x05fbbb (0xbbbc bytes) 000152[0] 1785391429.958089 mapread 0x0eb000 thru 0x0f4e0c (0x9e0d bytes) 000155[0] 1785391430.099802 write 0x01e000 thru 0x0211c5 (0x31c6 bytes) 000157[1] 1785391430.166725 mapread 0x037000 thru 0x0395fb (0x25fc bytes) 000159[0] 1785391430.267153 trunc from 0x1544fb to 0x2fb00a (0x1a6b10 bytes) 000161[1] 1785391430.412156 mapread 0x2c5000 thru 0x2d4823 (0xf824 bytes) 000162[0] 1785391430.559279 write 0x5e9000 thru 0x5ef85f (0x6860 bytes) 000164[0] 1785391430.619120 read 0x194000 thru 0x19916b (0x516c bytes) 000166[1] 1785391430.655947 mapread 0x501000 thru 0x50272c (0x172d bytes) 000167[0] 1785391430.731096 read 0x14b000 thru 0x1587f6 (0xd7f7 bytes) 000168[1] 1785391430.742756 write 0x26c000 thru 0x271e76 (0x5e77 bytes) 000171[1] 1785391430.808284 trunc from 0x5ef860 to 0x708efe (0x11969f bytes) 000172[1] 1785391430.840056 read 0x0cb000 thru 0x0d3fce (0x8fcf bytes) 000174[0] 1785391430.866508 mapwrite 0x0d2000 thru 0x0e137e (0xf37f bytes) 000176[1] 1785391431.136085 mapwrite 0x40b000 thru 0x4173b9 (0xc3ba bytes) 000180[1] 1785391431.381557 mapwrite 0x6b4000 thru 0x6c1f3a (0xdf3b bytes) 000181[0] 1785391431.506089 mapread 0x365000 thru 0x36b3c4 (0x63c5 bytes) 000183[0] 1785391431.563752 mapread 0x067000 thru 0x071b27 (0xab28 bytes) 000184[1] 1785391431.599553 write 0x62a000 thru 0x62ddfa (0x3dfb bytes) 000185[1] 1785391431.691971 mapwrite 0x63f000 thru 0x648700 (0x9701 bytes) 000186[1] 1785391431.805566 mapwrite 0x7d6000 thru 0x7dcc50 (0x6c51 bytes) 000190[0] 1785391431.968699 read 0x254000 thru 0x258595 (0x4596 bytes) 000192[1] 1785391432.012403 write 0x6e5000 thru 0x6f346f (0xe470 bytes) 000194[0] 1785391432.043709 mapwrite 0x0f9000 thru 0x0fb42f (0x2430 bytes) 000196[1] 1785391432.175337 mapread 0x3b8000 thru 0x3c025d (0x825e bytes) 000198[0] 1785391432.271242 trunc from 0x7dcc51 to 0x29a1bf (0x542a91 bytes) 000200[1] 1785391432.349887 trunc from 0x29a1c0 to 0x1289b2 (0x17180d bytes) 000202[0] 1785391432.417302 read 0x10a000 thru 0x1149f5 (0xa9f6 bytes) 000204[1] 1785391432.488063 trunc from 0x1289b3 to 0x0eac32 (0x3dd80 bytes) truncating to largest ever: 0x937fe3 000208[1] 1785391432.584937 trunc from 0x0eac33 to 0x937fe2 (0x84d3b0 bytes) 000209[0] 1785391432.622391 write 0x2bc000 thru 0x2bc7e5 (0x7e6 bytes) 000210[0] 1785391432.739707 write 0x0a9000 thru 0x0b662c (0xd62d bytes) 000214[1] 1785391432.798841 mapread 0x8c0000 thru 0x8c3767 (0x3768 bytes) 000215[0] 1785391432.845600 mapread 0x37e000 thru 0x387b7d (0x9b7e bytes) 000216[1] 1785391432.901149 mapread 0x60f000 thru 0x61ca04 (0xda05 bytes) 000217[0] 1785391432.927412 mapwrite 0x5a4000 thru 0x5b022d (0xc22e bytes) 000218[0] 1785391433.061912 mapwrite 0x53d000 thru 0x53f5a6 (0x25a7 bytes) 000219[0] 1785391433.099150 trunc from 0x937fe3 to 0x6a21d2 (0x295e10 bytes) 000223[1] 1785391433.150158 trunc from 0x6a21d3 to 0x13bdac (0x566426 bytes) 000224[0] 1785391433.200072 mapread 0x03d000 thru 0x03df61 (0xf62 bytes) 000225[0] 1785391433.228995 mapread 0x0ee000 thru 0x0fb701 (0xd702 bytes) 000227[0] 1785391433.291056 read 0x03a000 thru 0x0435b9 (0x95ba bytes) 000229[0] 1785391433.350824 trunc from 0x13bdad to 0x2ea634 (0x1ae888 bytes) 000233[1] 1785391433.523747 mapread 0x1b7000 thru 0x1be98c (0x798d bytes) 000235[1] 1785391433.578311 mapwrite 0x79f000 thru 0x7a7314 (0x8315 bytes) 000237[0] 1785391433.879635 mapwrite 0x4ce000 thru 0x4d235c (0x435d bytes) 000238[0] 1785391434.257491 mapwrite 0x987000 thru 0x995c1d (0xec1e bytes) 000240[1] 1785391434.623342 trunc from 0x995c1e to 0x7d2d16 (0x1c2f07 bytes) 000243[1] 1785391434.740413 write 0x363000 thru 0x36d91a (0xa91b bytes) 000245[1] 1785391434.814404 mapread 0x5bf000 thru 0x5cda54 (0xea55 bytes) 000246[1] 1785391434.927149 mapwrite 0x830000 thru 0x83693b (0x693c bytes) 000247[1] 1785391435.173114 mapwrite 0x308000 thru 0x3151ab (0xd1ac bytes) 000250[1] 1785391435.481430 mapwrite 0x61b000 thru 0x61f837 (0x4838 bytes) 000254[1] 1785391435.811689 trunc from 0x83693c to 0x67b0ea (0x1bb851 bytes) 000255[1] 1785391435.846478 mapwrite 0x1bf000 thru 0x1c44ae (0x54af bytes) 000256[0] 1785391436.077135 write 0x74c000 thru 0x74dfc1 (0x1fc2 bytes) 000257[1] 1785391436.168880 mapread 0x10c000 thru 0x117229 (0xb22a bytes) 000259[0] 1785391436.259454 mapwrite 0x2fa000 thru 0x2fd287 (0x3288 bytes) 000260[1] 1785391436.485503 mapread 0x486000 thru 0x48731f (0x1320 bytes) 000263[1] 1785391436.564201 trunc from 0x74dfc2 to 0x09bb9e (0x6b2423 bytes) 000264[0] 1785391436.690150 write 0x9ec000 thru 0x9f3830 (0x7831 bytes) 000265[0] 1785391436.780493 write 0x447000 thru 0x44b413 (0x4414 bytes) 000266[0] 1785391436.791772 write 0x979000 thru 0x9864d5 (0xd4d6 bytes) 000267[1] 1785391436.800739 mapwrite 0x7aa000 thru 0x7acf83 (0x2f84 bytes) 000268[0] 1785391437.019948 mapread 0x581000 thru 0x582091 (0x1092 bytes) 000271[1] 1785391437.099679 read 0x3df000 thru 0x3ecb62 (0xdb63 bytes) 000272[1] 1785391437.134215 mapread 0x497000 thru 0x4a3ee9 (0xceea bytes) 000274[1] 1785391437.208811 mapread 0x53b000 thru 0x544652 (0x9653 bytes) 000277[1] 1785391437.320644 mapread 0x27a000 thru 0x286f8e (0xcf8f bytes) 000282[0] 1785391437.409496 read 0x470000 thru 0x47f6c7 (0xf6c8 bytes) 000284[0] 1785391437.420474 trunc from 0x9f3831 to 0x92ddba (0xc5a76 bytes) 000285[0] 1785391437.577039 mapwrite 0x427000 thru 0x429c84 (0x2c85 bytes) 000286[1] 1785391437.779968 trunc from 0x92ddbb to 0x535e50 (0x3f7f6a bytes) 000287[1] 1785391437.883374 write 0x32e000 thru 0x33684c (0x884d bytes) 000292[1] 1785391437.990409 mapwrite 0x2ea000 thru 0x2f63f5 (0xc3f6 bytes) 000294[0] 1785391438.177262 mapwrite 0x5d8000 thru 0x5e169b (0x969c bytes) 000299[1] 1785391438.591887 mapread 0x5c0000 thru 0x5cd817 (0xd818 bytes) 000300[1] 1785391438.684539 mapwrite 0x41d000 thru 0x42c1e1 (0xf1e2 bytes) 000301[0] 1785391438.808583 mapread 0x528000 thru 0x533cca (0xbccb bytes) 000305[1] 1785391438.847613 trunc from 0x5e169c to 0x203f47 (0x3dd754 bytes) 000306[0] 1785391438.896826 read 0x171000 thru 0x172874 (0x1875 bytes) 000312[1] 1785391438.978320 mapread 0x107000 thru 0x1075a9 (0x5aa bytes) 000315[0] 1785391438.984470 trunc from 0x203f48 to 0x74fb93 (0x54bc4c bytes) 000316[1] 1785391439.028921 read 0x185000 thru 0x18cf8d (0x7f8e bytes) 000319[1] 1785391439.108462 trunc from 0x74fb94 to 0x412af2 (0x33d0a1 bytes) 000321[1] 1785391439.165289 read 0x070000 thru 0x074779 (0x477a bytes) 000327[1] 1785391439.216937 write 0x123000 thru 0x12977f (0x6780 bytes) 000329[0] 1785391439.223674 trunc from 0x412af3 to 0x4534cf (0x409dd bytes) 000330[1] 1785391439.267339 mapwrite 0x6d7000 thru 0x6daeb8 (0x3eb9 bytes) 000332[0] 1785391439.421470 trunc from 0x6daeb9 to 0x1fb7f8 (0x4df6c0 bytes) 000335[1] 1785391439.463347 read 0x07f000 thru 0x08e078 (0xf079 bytes) 000338[1] 1785391439.522280 mapwrite 0x54e000 thru 0x550854 (0x2855 bytes) 000339[0] 1785391439.677967 read 0x0bf000 thru 0x0c8062 (0x9063 bytes) 000341[0] 1785391439.712975 mapwrite 0x69d000 thru 0x69e47d (0x147e bytes) 000343[1] 1785391439.809923 trunc from 0x69e47e to 0x125e49 (0x578634 bytes) 000344[1] 1785391439.856262 trunc from 0x125e4a to 0x8c61c2 (0x7a0379 bytes) 000349[0] 1785391439.895709 trunc from 0x8c61c3 to 0x698af9 (0x22d6c9 bytes) 000350[1] 1785391439.929512 read 0x315000 thru 0x321404 (0xc405 bytes) 000352[1] 1785391439.957848 trunc from 0x698afa to 0x90ae21 (0x272328 bytes) 000353[1] 1785391440.015628 mapread 0x5b5000 thru 0x5b9319 (0x431a bytes) 000354[0] 1785391440.059012 mapread 0x1b0000 thru 0x1bde58 (0xde59 bytes) 000355[1] 1785391440.108833 write 0x574000 thru 0x575132 (0x1133 bytes) 000360[0] 1785391440.183122 mapread 0x36e000 thru 0x37bb32 (0xdb33 bytes) 000363[0] 1785391440.253825 mapwrite 0x5d1000 thru 0x5d33f5 (0x23f6 bytes) 000364[0] 1785391440.351425 mapread 0x0c4000 thru 0x0ceeb4 (0xaeb5 bytes) 000366[0] 1785391440.411324 mapwrite 0x55a000 thru 0x55ebf2 (0x4bf3 bytes) 000367[1] 1785391440.503700 write 0x281000 thru 0x287729 (0x672a bytes) 000369[0] 1785391440.557519 mapread 0x475000 thru 0x47c9f1 (0x79f2 bytes) 000371[0] 1785391440.624113 read 0x1b5000 thru 0x1b8225 (0x3226 bytes) 000372[0] 1785391440.634909 write 0x949000 thru 0x94bfda (0x2fdb bytes) 000376[1] 1785391440.720369 read 0x195000 thru 0x19b49b (0x649c bytes) 000377[1] 1785391440.768643 mapwrite 0x96f000 thru 0x9761f7 (0x71f8 bytes) 000380[0] 1785391441.032299 mapread 0x88f000 thru 0x896235 (0x7236 bytes) 000381[0] 1785391441.127269 read 0x140000 thru 0x149065 (0x9066 bytes) 000384[1] 1785391441.139893 read 0x350000 thru 0x3509a1 (0x9a2 bytes) 000385[1] 1785391441.159492 mapwrite 0x0ac000 thru 0x0bb666 (0xf667 bytes) 000386[1] 1785391441.421866 write 0x40f000 thru 0x415a14 (0x6a15 bytes) 000389[0] 1785391441.491423 read 0x6f8000 thru 0x706683 (0xe684 bytes) 000390[1] 1785391441.555591 mapwrite 0x4b0000 thru 0x4b9b6e (0x9b6f bytes) 000391[1] 1785391441.756733 mapread 0x3de000 thru 0x3e1ca3 (0x3ca4 bytes) 000392[1] 1785391441.821446 mapread 0x558000 thru 0x56282d (0xa82e bytes) 000395[1] 1785391441.980068 trunc from 0x9761f8 to 0x929864 (0x4c993 bytes) 000396[0] 1785391442.019551 read 0x247000 thru 0x248bb0 (0x1bb1 bytes) 000400[1] 1785391442.092719 write 0x6d4000 thru 0x6e077f (0xc780 bytes) 000401[1] 1785391442.193114 mapwrite 0x246000 thru 0x246b85 (0xb86 bytes) 000402[1] 1785391442.313631 mapwrite 0x83c000 thru 0x841157 (0x5158 bytes) 000403[0] 1785391442.459626 mapread 0x831000 thru 0x837db8 (0x6db9 bytes) 000404[1] 1785391442.546957 write 0x508000 thru 0x50b140 (0x3141 bytes) 000407[0] 1785391442.605996 read 0x14e000 thru 0x150054 (0x2055 bytes) 000410[1] 1785391442.683453 mapwrite 0x9ee000 thru 0x9fc024 (0xe025 bytes) 000411[0] 1785391442.975442 mapwrite 0x342000 thru 0x34568a (0x368b bytes) 000413[0] 1785391443.300785 read 0x9d6000 thru 0x9d8ba9 (0x2baa bytes) 000414[0] 1785391443.311580 mapwrite 0x255000 thru 0x25c7eb (0x77ec bytes) 000416[0] 1785391443.596817 mapwrite 0x8bb000 thru 0x8c7975 (0xc976 bytes) 000419[0] 1785391443.775653 trunc from 0x9fc025 to 0x0aba7b (0x9505a9 bytes) 000420[0] 1785391443.831270 mapread 0x00e000 thru 0x012526 (0x4527 bytes) truncating to largest ever: 0x94fa28 000423[0] 1785391443.970224 trunc from 0x0aba7c to 0x94fa27 (0x8a3fac bytes) 000424[1] 1785391444.062845 read 0x1a1000 thru 0x1a3d5c (0x2d5d bytes) 000425[0] 1785391444.112346 mapread 0x855000 thru 0x863115 (0xe116 bytes) 000426[0] 1785391444.201139 mapwrite 0x91a000 thru 0x91b633 (0x1634 bytes) 000427[1] 1785391444.358343 mapread 0x1df000 thru 0x1e7323 (0x8324 bytes) 000429[0] 1785391444.447066 write 0x531000 thru 0x53e646 (0xd647 bytes) 000431[0] 1785391444.494970 read 0x4d0000 thru 0x4d9a4b (0x9a4c bytes) 000433[0] 1785391444.531627 trunc from 0x94fa28 to 0x2b5343 (0x69a6e4 bytes) 000435[1] 1785391444.624458 read 0x1c4000 thru 0x1c554f (0x1550 bytes) 000437[0] 1785391444.765910 write 0x0d1000 thru 0x0d52d5 (0x42d6 bytes) 000438[0] 1785391444.805672 mapwrite 0x337000 thru 0x343775 (0xc776 bytes) 000439[1] 1785391445.151959 mapread 0x1a1000 thru 0x1af17e (0xe17f bytes) 000442[0] 1785391445.446139 trunc from 0x343776 to 0x49e4aa (0x15ad35 bytes) 000443[1] 1785391445.585786 mapwrite 0x456000 thru 0x45a246 (0x4247 bytes) 000447[1] 1785391445.839066 write 0x08b000 thru 0x0926c8 (0x76c9 bytes) 000449[0] 1785391445.853836 mapread 0x261000 thru 0x26fa63 (0xea64 bytes) 000454[0] 1785391445.959317 mapread 0x015000 thru 0x01f6fd (0xa6fe bytes) 000463[0] 1785391446.007871 mapread 0x260000 thru 0x26a915 (0xa916 bytes) 000467[1] 1785391446.012951 read 0x226000 thru 0x2292e6 (0x32e7 bytes) 000468[1] 1785391446.031646 read 0x0b4000 thru 0x0be1d3 (0xa1d4 bytes) 000469[1] 1785391446.055879 write 0x3c7000 thru 0x3d47d9 (0xd7da bytes) 000472[1] 1785391446.137981 mapread 0x39a000 thru 0x39f87a (0x587b bytes) 000473[0] 1785391446.175043 write 0x013000 thru 0x01e18a (0xb18b bytes) 000474[0] 1785391446.235236 mapread 0x0dc000 thru 0x0e617d (0xa17e bytes) 000476[1] 1785391446.309539 read 0x2f1000 thru 0x30093f (0xf940 bytes) 000477[1] 1785391446.351256 write 0x23f000 thru 0x24ed5a (0xfd5b bytes) 000479[0] 1785391446.429433 read 0x29c000 thru 0x2a6188 (0xa189 bytes) 000481[1] 1785391446.494413 mapread 0x421000 thru 0x427952 (0x6953 bytes) 000484[0] 1785391446.528642 mapread 0x465000 thru 0x470156 (0xb157 bytes) 000492[0] 1785391446.625113 trunc from 0x49e4ab to 0x56ea26 (0xd057c bytes) 000493[1] 1785391446.718405 mapread 0x156000 thru 0x16147f (0xb480 bytes) 000496[1] 1785391446.844715 read 0x1d1000 thru 0x1dece8 (0xdce9 bytes) 000500[0] 1785391446.889842 write 0x2a2000 thru 0x2a6062 (0x4063 bytes) All operations completed A-OK! PASS 16a (71s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:04:14 (1785391454) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.645642 s, 13.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.652549 s, 12.9 MB/s Chance of close/open is 1 in 50 Seed set to 1458 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000007[0] 1785391459.738324 write 0x96dc0f thru 0x96e672 (0xa64 bytes) 000009[1] 1785391459.837940 write 0x245fcd thru 0x24778e (0x17c2 bytes) 000010[0] 1785391459.951290 read 0x62b644 thru 0x62d515 (0x1ed2 bytes) truncating to largest ever: 0x8fe3f0 000011[0] 1785391460.072963 trunc from 0x96e673 to 0x8fe3ef (0x70283 bytes) 000012[0] 1785391460.191705 mapread 0x5ef923 thru 0x5f0e0d (0x14eb bytes) 000013[1] 1785391460.208115 write 0x950f3b thru 0x9518b3 (0x979 bytes) 000016[0] 1785391460.353301 read 0x01ee9e thru 0x01f176 (0x2d9 bytes) 000021[1] 1785391460.544323 read 0x6501f6 thru 0x6519e2 (0x17ed bytes) 000022[1] 1785391460.616808 mapwrite 0x2a0893 thru 0x2a1f52 (0x16c0 bytes) 000024[1] 1785391460.948432 write 0x30c958 thru 0x30d8a5 (0xf4e bytes) 000026[1] 1785391461.025940 write 0x0dcbc7 thru 0x0de180 (0x15ba bytes) 000027[1] 1785391461.056937 write 0x60455c thru 0x606151 (0x1bf6 bytes) 000028[0] 1785391461.112987 mapwrite 0x2e234b thru 0x2e375f (0x1415 bytes) 000030[1] 1785391461.269235 write 0x69bddf thru 0x69d530 (0x1752 bytes) 000031[0] 1785391461.335733 read 0x52ce91 thru 0x52ee1b (0x1f8b bytes) 000033[1] 1785391461.482613 write 0x112207 thru 0x112e13 (0xc0d bytes) 000034[0] 1785391461.511048 mapread 0x011b77 thru 0x01359f (0x1a29 bytes) 000035[1] 1785391461.589374 mapwrite 0x6755df thru 0x675bbf (0x5e1 bytes) 000042[0] 1785391461.935310 mapread 0x38a880 thru 0x38b2ed (0xa6e bytes) 000044[0] 1785391462.076995 trunc from 0x9518b4 to 0x7f991c (0x157f97 bytes) 000045[0] 1785391462.246327 trunc from 0x7f991d to 0x1cb89a (0x62e082 bytes) 000049[1] 1785391462.536801 mapwrite 0x3af44c thru 0x3af473 (0x28 bytes) 000051[1] 1785391462.857721 mapread 0x19a6e7 thru 0x19a945 (0x25f bytes) truncating to largest ever: 0x98fa0a 000054[1] 1785391462.995900 trunc from 0x3af474 to 0x98fa09 (0x5e0596 bytes) 000055[1] 1785391463.068786 read 0x410fb2 thru 0x4122b8 (0x1307 bytes) 000056[1] 1785391463.109345 trunc from 0x98fa0a to 0x103c03 (0x88be06 bytes) 000060[1] 1785391463.213433 read 0x08a337 thru 0x08b3a4 (0x106e bytes) 000063[1] 1785391463.281079 mapwrite 0x177c00 thru 0x178049 (0x44a bytes) 000064[0] 1785391463.358563 write 0x05915f thru 0x05afa2 (0x1e44 bytes) 000065[1] 1785391463.392403 read 0x075afe thru 0x076a5e (0xf61 bytes) 000066[1] 1785391463.441206 mapread 0x0960e9 thru 0x096fec (0xf04 bytes) 000067[1] 1785391463.451476 read 0x055ad7 thru 0x056597 (0xac1 bytes) 000068[1] 1785391463.452690 read 0x08a3a3 thru 0x08acfd (0x95b bytes) 000069[0] 1785391463.462638 mapwrite 0x9a3b49 thru 0x9a54e7 (0x199f bytes) 000070[1] 1785391463.559023 write 0x135b50 thru 0x13615f (0x610 bytes) 000072[0] 1785391463.658861 write 0x61c009 thru 0x61c6dd (0x6d5 bytes) 000076[0] 1785391463.710248 read 0x316929 thru 0x31791e (0xff6 bytes) 000077[1] 1785391463.774714 trunc from 0x9a54e8 to 0x7d7a82 (0x1cda65 bytes) 000079[0] 1785391463.849846 mapread 0x3783bf thru 0x379fcf (0x1c11 bytes) 000082[0] 1785391464.039418 mapwrite 0x9aa67e thru 0x9ab85e (0x11e1 bytes) 000085[0] 1785391464.339630 mapwrite 0x8d3b21 thru 0x8d4d63 (0x1243 bytes) 000088[1] 1785391464.466148 trunc from 0x9ab85f to 0x46f81b (0x53c043 bytes) 000091[1] 1785391464.593568 mapread 0x1130ea thru 0x1140dc (0xff3 bytes) truncating to largest ever: 0x99e067 000093[1] 1785391464.632408 trunc from 0x46f81c to 0x99e066 (0x52e84b bytes) 000095[1] 1785391464.653405 mapread 0x1c637a thru 0x1c80c3 (0x1d4a bytes) 000098[1] 1785391464.698071 trunc from 0x99e067 to 0x657991 (0x3466d5 bytes) 000100[1] 1785391464.751590 write 0x15335c thru 0x153ed0 (0xb75 bytes) 000102[0] 1785391464.819715 mapread 0x270e37 thru 0x2717ba (0x984 bytes) 000103[0] 1785391464.951706 mapwrite 0x350211 thru 0x351313 (0x1103 bytes) 000105[0] 1785391465.210621 mapread 0x089c05 thru 0x08ba81 (0x1e7d bytes) 000108[0] 1785391465.434521 mapread 0x3fdff2 thru 0x3ff17f (0x118e bytes) 000111[1] 1785391465.647077 mapwrite 0x8e5715 thru 0x8e5aa6 (0x392 bytes) 000112[1] 1785391465.864033 read 0x8ab528 thru 0x8ac4fe (0xfd7 bytes) 000115[1] 1785391465.966040 trunc from 0x8e5aa7 to 0x763439 (0x18266d bytes) 000116[1] 1785391466.002543 trunc from 0x76343a to 0x5422bd (0x22117c bytes) 000117[0] 1785391466.090605 mapwrite 0x2ff023 thru 0x300261 (0x123f bytes) 000121[0] 1785391466.432581 trunc from 0x5422be to 0x6e3e92 (0x1a1bd5 bytes) 000122[1] 1785391466.561552 mapread 0x67bb02 thru 0x67c45d (0x95c bytes) 000123[0] 1785391466.712067 read 0x55a02c thru 0x55a16e (0x143 bytes) 000124[1] 1785391466.738247 write 0x5e160c thru 0x5e20d1 (0xac6 bytes) 000129[0] 1785391467.142555 mapwrite 0x4324f4 thru 0x432638 (0x145 bytes) 000132[0] 1785391467.394818 read 0x34aa89 thru 0x34af44 (0x4bc bytes) 000135[0] 1785391467.447989 mapread 0x1141a5 thru 0x115c48 (0x1aa4 bytes) 000136[1] 1785391467.475636 trunc from 0x6e3e93 to 0x2da747 (0x40974b bytes) 000139[0] 1785391467.644420 write 0x470227 thru 0x47133c (0x1116 bytes) 000141[0] 1785391467.728448 read 0x468bbe thru 0x4698f0 (0xd33 bytes) 000151[0] 1785391468.129458 mapread 0x0caa39 thru 0x0cba96 (0x105e bytes) 000152[1] 1785391468.230360 mapwrite 0x8d1353 thru 0x8d305a (0x1d08 bytes) 000153[0] 1785391468.597671 trunc from 0x8d305b to 0x6c1d0c (0x21134e bytes) 000154[1] 1785391468.698652 trunc from 0x6c1d0d to 0x2e332e (0x3de9de bytes) 000155[0] 1785391468.746719 mapwrite 0x0cfc8e thru 0x0d05eb (0x95e bytes) 000156[1] 1785391468.900691 trunc from 0x2e332f to 0x1ba080 (0x1292ae bytes) 000157[0] 1785391468.977940 trunc from 0x1ba081 to 0x3470a6 (0x18d026 bytes) 000159[1] 1785391469.076611 mapwrite 0x0c88ce thru 0x0c928e (0x9c1 bytes) 000164[0] 1785391469.370966 mapwrite 0x6224fa thru 0x623bee (0x16f5 bytes) 000165[0] 1785391469.572421 mapwrite 0x8c2773 thru 0x8c39fe (0x128c bytes) 000166[1] 1785391469.718672 mapread 0x4f2d4c thru 0x4f3c58 (0xf0d bytes) 000167[1] 1785391469.774109 trunc from 0x8c39ff to 0x3af7d5 (0x514229 bytes) 000172[0] 1785391469.946409 mapwrite 0x46dfc3 thru 0x46ec03 (0xc41 bytes) 000175[1] 1785391470.177074 write 0x919541 thru 0x91ae3e (0x18fe bytes) 000178[1] 1785391470.278463 trunc from 0x91ae3f to 0x2b41ce (0x666c70 bytes) 000180[1] 1785391470.376236 mapwrite 0x9bc6d5 thru 0x9bd5c1 (0xeed bytes) 000181[0] 1785391470.611831 read 0x6d2dc6 thru 0x6d45d7 (0x1812 bytes) 000183[0] 1785391470.762398 mapwrite 0x7ffd1f thru 0x8002f7 (0x5d9 bytes) 000184[1] 1785391471.246562 mapwrite 0x13cda7 thru 0x13dcd5 (0xf2f bytes) 000185[0] 1785391471.499378 read 0x8d9c49 thru 0x8da8ae (0xc66 bytes) 000187[0] 1785391471.554730 write 0x90904a thru 0x90a8e8 (0x189f bytes) 000188[1] 1785391471.625377 mapwrite 0x17d75b thru 0x17ec34 (0x14da bytes) 000190[0] 1785391471.867416 mapwrite 0x9018f1 thru 0x902618 (0xd28 bytes) 000192[0] 1785391472.054517 mapread 0x0e37f3 thru 0x0e484e (0x105c bytes) 000193[1] 1785391472.153341 read 0x700649 thru 0x701b4d (0x1505 bytes) 000197[1] 1785391472.314776 mapwrite 0x7dccbb thru 0x7de170 (0x14b6 bytes) 000198[0] 1785391472.622803 write 0x93c0ad thru 0x93c199 (0xed bytes) 000199[0] 1785391472.663091 trunc from 0x9bd5c2 to 0x93c166 (0x8145b bytes) 000201[0] 1785391472.792351 write 0x505335 thru 0x505e70 (0xb3c bytes) 000203[1] 1785391472.845485 mapwrite 0x6ce343 thru 0x6cf258 (0xf16 bytes) 000214[0] 1785391473.099623 mapwrite 0x4c08ba thru 0x4c11ed (0x934 bytes) 000215[0] 1785391473.151968 read 0x80a1d3 thru 0x80a518 (0x346 bytes) 000217[1] 1785391473.185323 mapwrite 0x2f7b71 thru 0x2f81ee (0x67e bytes) 000219[0] 1785391473.296645 read 0x602d31 thru 0x604c11 (0x1ee1 bytes) 000220[1] 1785391473.318748 trunc from 0x93c167 to 0x13b248 (0x800f1e bytes) 000222[0] 1785391473.379767 write 0x2e282c thru 0x2e3dd1 (0x15a6 bytes) 000223[1] 1785391473.400700 write 0x8112c3 thru 0x8113f8 (0x136 bytes) 000228[0] 1785391473.490001 write 0x7a24d3 thru 0x7a4021 (0x1b4f bytes) 000229[1] 1785391473.526869 write 0x021936 thru 0x021ddf (0x4aa bytes) 000231[0] 1785391473.575568 mapread 0x32051c thru 0x320d4c (0x831 bytes) 000232[0] 1785391473.611469 write 0x99927d thru 0x99959e (0x322 bytes) 000240[0] 1785391473.693127 mapread 0x0bc159 thru 0x0bcbf6 (0xa9e bytes) 000241[0] 1785391473.716037 trunc from 0x99959f to 0x835aab (0x163af3 bytes) 000247[0] 1785391473.840070 mapwrite 0x489a4b thru 0x48a81a (0xdd0 bytes) 000249[0] 1785391473.895407 write 0x85ede5 thru 0x85fe25 (0x1041 bytes) 000250[1] 1785391473.914326 write 0x863756 thru 0x864e7f (0x172a bytes) 000251[1] 1785391473.940030 write 0x2aa57d thru 0x2ac14d (0x1bd1 bytes) 000256[1] 1785391474.012749 trunc from 0x864e80 to 0x20b6c2 (0x6597bd bytes) 000257[1] 1785391474.071013 write 0x564d47 thru 0x56602b (0x12e5 bytes) 000263[1] 1785391474.201238 mapread 0x3c2527 thru 0x3c313a (0xc14 bytes) 000264[1] 1785391474.234345 mapread 0x12de17 thru 0x12edd2 (0xfbc bytes) 000265[0] 1785391474.294054 trunc from 0x56602c to 0x6df8c2 (0x179897 bytes) 000266[0] 1785391474.389679 mapread 0x062713 thru 0x064280 (0x1b6e bytes) 000268[1] 1785391474.407530 read 0x4acba7 thru 0x4add16 (0x1170 bytes) 000269[0] 1785391474.498964 mapread 0x17249a thru 0x172be0 (0x747 bytes) 000270[0] 1785391474.522748 write 0x94903d thru 0x949465 (0x429 bytes) 000273[0] 1785391474.568085 mapwrite 0x5e46d0 thru 0x5e4bad (0x4de bytes) 000274[0] 1785391474.705987 trunc from 0x949466 to 0x298fe8 (0x6b047d bytes) 000275[1] 1785391474.766836 write 0x724223 thru 0x724c54 (0xa32 bytes) 000277[0] 1785391474.808739 read 0x1f1cb7 thru 0x1f39b6 (0x1d00 bytes) 000278[1] 1785391474.827054 write 0x2db887 thru 0x2dcf54 (0x16ce bytes) 000279[1] 1785391474.851119 mapwrite 0x6c5847 thru 0x6c62c8 (0xa82 bytes) 000281[1] 1785391474.932346 read 0x24ea19 thru 0x24f1f7 (0x7df bytes) 000282[1] 1785391474.960594 trunc from 0x724c55 to 0x95b18d (0x236539 bytes) 000288[1] 1785391475.057995 trunc from 0x95b18e to 0x8e24cb (0x78cc2 bytes) 000291[1] 1785391475.188666 read 0x63f5f7 thru 0x640d26 (0x1730 bytes) 000293[0] 1785391475.240024 read 0x43c935 thru 0x43df46 (0x1612 bytes) 000294[1] 1785391475.322687 mapwrite 0x1d58a0 thru 0x1d5f6e (0x6cf bytes) 000295[0] 1785391475.473155 mapwrite 0x4adc54 thru 0x4af793 (0x1b40 bytes) 000301[0] 1785391475.862987 write 0x2cdff6 thru 0x2ce2bc (0x2c7 bytes) 000303[0] 1785391475.930623 trunc from 0x8e24cc to 0x20de88 (0x6d4643 bytes) 000305[0] 1785391475.990390 trunc from 0x20de89 to 0x894831 (0x6869a9 bytes) 000306[0] 1785391476.041207 mapread 0x1bc3d9 thru 0x1be26e (0x1e96 bytes) 000307[1] 1785391476.076916 mapwrite 0x1fad94 thru 0x1fbf67 (0x11d4 bytes) 000308[0] 1785391476.312328 mapwrite 0x45a4e5 thru 0x45a75b (0x277 bytes) 000310[1] 1785391476.489511 trunc from 0x894832 to 0x0c07fc (0x7d4035 bytes) 000311[0] 1785391476.558447 write 0x2b74ca thru 0x2b8d71 (0x18a8 bytes) 000313[1] 1785391476.601028 trunc from 0x2b8d72 to 0x56001c (0x2a72ab bytes) 000314[1] 1785391476.632054 mapread 0x458795 thru 0x458959 (0x1c5 bytes) 000315[1] 1785391476.666473 trunc from 0x56001d to 0x94b340 (0x3eb324 bytes) 000316[1] 1785391476.693945 write 0x459250 thru 0x45acac (0x1a5d bytes) 000317[0] 1785391476.700977 mapwrite 0x609406 thru 0x60b3f5 (0x1ff0 bytes) 000318[0] 1785391476.824711 trunc from 0x94b341 to 0x7355cf (0x215d71 bytes) 000319[0] 1785391476.882496 mapwrite 0x3dbe96 thru 0x3dcde6 (0xf51 bytes) 000323[1] 1785391476.989463 mapread 0x00b701 thru 0x00b8eb (0x1eb bytes) 000330[1] 1785391477.053494 mapwrite 0x78f6e5 thru 0x790629 (0xf45 bytes) 000331[1] 1785391477.130016 read 0x01e05d thru 0x01fc9a (0x1c3e bytes) 000333[1] 1785391477.155218 read 0x3e21c7 thru 0x3e22cd (0x107 bytes) 000334[0] 1785391477.159700 trunc from 0x79062a to 0x4b5dcb (0x2da85e bytes) 000335[0] 1785391477.192651 mapwrite 0x0c2cf4 thru 0x0c2e2c (0x139 bytes) 000339[0] 1785391477.312346 trunc from 0x4b5dcc to 0x639d26 (0x183f5b bytes) 000340[1] 1785391477.376385 mapread 0x52a404 thru 0x52a4c6 (0xc3 bytes) 000342[0] 1785391477.457474 write 0x77d6fe thru 0x77f152 (0x1a55 bytes) 000343[1] 1785391477.537986 mapwrite 0x5a8e0f thru 0x5aac6f (0x1e61 bytes) 000345[1] 1785391477.776043 mapwrite 0x7b050c thru 0x7b232f (0x1e24 bytes) 000346[1] 1785391477.903824 write 0x30308a thru 0x3030a8 (0x1f bytes) 000347[0] 1785391477.931948 write 0x11b6b7 thru 0x11bfbf (0x909 bytes) 000349[0] 1785391477.980971 write 0x72e7fb thru 0x72ebfd (0x403 bytes) 000351[0] 1785391478.012479 write 0x9cc03d thru 0x9cde52 (0x1e16 bytes) 000352[0] 1785391478.032430 mapread 0x08171d thru 0x081ff2 (0x8d6 bytes) 000354[1] 1785391478.054952 mapwrite 0x351b8a thru 0x351d0e (0x185 bytes) 000355[0] 1785391478.279516 write 0x9bd643 thru 0x9bd733 (0xf1 bytes) 000356[0] 1785391478.296609 read 0x126377 thru 0x12759f (0x1229 bytes) 000357[1] 1785391478.309864 mapwrite 0x3c7d48 thru 0x3c90b8 (0x1371 bytes) 000360[1] 1785391478.493915 write 0x3781ee thru 0x378f15 (0xd28 bytes) 000361[1] 1785391478.509284 mapwrite 0x961067 thru 0x9612b7 (0x251 bytes) 000362[0] 1785391478.566479 read 0x6b6437 thru 0x6b6485 (0x4f bytes) 000366[1] 1785391478.609291 read 0x25c74b thru 0x25db70 (0x1426 bytes) 000368[0] 1785391478.636294 mapread 0x24b82e thru 0x24bc4b (0x41e bytes) 000369[0] 1785391478.655732 write 0x36f777 thru 0x37126d (0x1af7 bytes) 000370[1] 1785391478.684443 trunc from 0x9cde53 to 0x60d71c (0x3c0736 bytes) 000372[0] 1785391478.736821 mapwrite 0x4c18ec thru 0x4c28bd (0xfd2 bytes) 000375[1] 1785391478.825004 read 0x5e3237 thru 0x5e3584 (0x34e bytes) 000377[1] 1785391478.836066 read 0x3b3120 thru 0x3b4799 (0x167a bytes) 000378[1] 1785391478.863496 trunc from 0x60d71d to 0x84bfa0 (0x23e884 bytes) 000383[0] 1785391478.941792 trunc from 0x84bfa1 to 0x440da2 (0x40b1fe bytes) 000384[1] 1785391478.999740 mapwrite 0x1bc920 thru 0x1bd2b9 (0x99a bytes) 000385[0] 1785391479.075610 write 0x68d375 thru 0x68eb48 (0x17d4 bytes) 000386[0] 1785391479.089108 write 0x721b53 thru 0x7236b5 (0x1b63 bytes) 000388[0] 1785391479.111992 mapread 0x47009d thru 0x470e44 (0xda8 bytes) 000389[1] 1785391479.172765 read 0x56a478 thru 0x56bc4a (0x17d3 bytes) 000392[0] 1785391479.215635 mapread 0x165493 thru 0x1665c1 (0x112f bytes) 000395[1] 1785391479.291924 mapread 0x6b7772 thru 0x6b84f6 (0xd85 bytes) 000396[1] 1785391479.311814 trunc from 0x7236b6 to 0x1e10be (0x5425f7 bytes) 000400[1] 1785391479.602828 mapwrite 0x23617a thru 0x236f3e (0xdc5 bytes) 000401[0] 1785391479.716021 read 0x0e030f thru 0x0e08ae (0x5a0 bytes) 000404[0] 1785391479.863501 mapread 0x014643 thru 0x0147a0 (0x15e bytes) 000406[0] 1785391479.893477 mapread 0x0cefa1 thru 0x0d0828 (0x1888 bytes) 000407[1] 1785391479.917905 write 0x6d9842 thru 0x6daec9 (0x1688 bytes) 000408[1] 1785391479.921138 write 0x3b0381 thru 0x3b142a (0x10aa bytes) 000409[1] 1785391479.940406 read 0x2f2255 thru 0x2f2c5c (0xa08 bytes) 000411[1] 1785391479.976321 write 0x41c77e thru 0x41cc6f (0x4f2 bytes) 000413[0] 1785391479.996566 write 0x0e8ef7 thru 0x0eacbc (0x1dc6 bytes) 000416[0] 1785391480.129143 write 0x4b0150 thru 0x4b034b (0x1fc bytes) 000418[0] 1785391480.169775 write 0x963520 thru 0x963974 (0x455 bytes) 000424[1] 1785391480.205799 write 0x275ce8 thru 0x27727a (0x1593 bytes) 000429[0] 1785391480.312210 mapwrite 0x5dd892 thru 0x5ddeb6 (0x625 bytes) 000430[1] 1785391480.404360 read 0x658214 thru 0x658b0c (0x8f9 bytes) 000431[0] 1785391480.428262 write 0x5fc2fa thru 0x5fcf2c (0xc33 bytes) 000433[1] 1785391480.512127 mapwrite 0x5a9767 thru 0x5aaddc (0x1676 bytes) 000437[0] 1785391480.648911 read 0x1ec8ba thru 0x1ee281 (0x19c8 bytes) 000438[1] 1785391480.686923 mapwrite 0x61410c thru 0x615cd9 (0x1bce bytes) 000439[0] 1785391480.828611 mapread 0x2fb984 thru 0x2fbd47 (0x3c4 bytes) 000441[0] 1785391480.856278 mapread 0x89eb15 thru 0x8a053d (0x1a29 bytes) 000445[1] 1785391480.928119 read 0x5a2611 thru 0x5a2f81 (0x971 bytes) 000447[1] 1785391480.936331 write 0x76e216 thru 0x76fb08 (0x18f3 bytes) 000448[0] 1785391480.948037 mapread 0x2ae1b9 thru 0x2ae510 (0x358 bytes) 000449[1] 1785391480.970333 mapwrite 0x7cbe4b thru 0x7cdd52 (0x1f08 bytes) 000450[0] 1785391481.139278 write 0x279b3a thru 0x27a142 (0x609 bytes) 000451[1] 1785391481.220822 mapread 0x8afb3a thru 0x8afdd1 (0x298 bytes) 000452[0] 1785391481.327907 mapwrite 0x9c59b8 thru 0x9c78b2 (0x1efb bytes) 000453[1] 1785391481.664687 mapread 0x838220 thru 0x8385f2 (0x3d3 bytes) 000455[0] 1785391481.767074 read 0x94678f thru 0x946f76 (0x7e8 bytes) 000456[0] 1785391481.828972 mapwrite 0x10c2b4 thru 0x10cb2b (0x878 bytes) 000459[1] 1785391482.048290 write 0x5e2c48 thru 0x5e4bad (0x1f66 bytes) 000460[0] 1785391482.127554 mapwrite 0x02df1a thru 0x02e18f (0x276 bytes) 000461[1] 1785391482.228954 trunc from 0x9c78b3 to 0x2d214f (0x6f5763 bytes) 000463[1] 1785391482.287765 write 0x6686cf thru 0x669172 (0xaa4 bytes) 000464[1] 1785391482.292990 read 0x12ef8d thru 0x130510 (0x1584 bytes) 000467[0] 1785391482.331438 mapwrite 0x45882c thru 0x459ea0 (0x1675 bytes) 000468[1] 1785391482.406457 mapread 0x303290 thru 0x304405 (0x1176 bytes) 000470[1] 1785391482.472494 mapread 0x130622 thru 0x131361 (0xd40 bytes) 000473[1] 1785391482.542919 read 0x043f08 thru 0x045785 (0x187e bytes) 000474[0] 1785391482.571501 read 0x488738 thru 0x48935a (0xc23 bytes) 000475[0] 1785391482.590595 mapread 0x55a4b7 thru 0x55c1df (0x1d29 bytes) 000480[1] 1785391482.674800 mapwrite 0x08d7ce thru 0x08ea2f (0x1262 bytes) 000481[0] 1785391482.747262 mapread 0x559520 thru 0x55a02b (0xb0c bytes) 000483[0] 1785391482.786983 mapread 0x1f5229 thru 0x1f5378 (0x150 bytes) 000489[0] 1785391482.909124 mapwrite 0x9e18b8 thru 0x9e24f6 (0xc3f bytes) 000490[1] 1785391483.131647 mapwrite 0x2f3989 thru 0x2f4735 (0xdad bytes) 000491[0] 1785391483.381361 read 0x340526 thru 0x341f0f (0x19ea bytes) 000493[1] 1785391483.485915 mapread 0x194300 thru 0x1944f9 (0x1fa bytes) 000496[1] 1785391483.586397 write 0x1f93f7 thru 0x1fab65 (0x176f bytes) 000497[1] 1785391483.641909 mapread 0x01b546 thru 0x01b5e1 (0x9c bytes) 000500[1] 1785391483.734934 mapread 0x70faf0 thru 0x7111c6 (0x16d7 bytes) All operations completed A-OK! PASS 16b (36s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:04:50 (1785391490) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:04:56 (1785391496) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.801295 s, 10.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.482925 s, 17.4 MB/s ldlm.namespaces.MGC192.168.202.132@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff965984c46000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff965991e9f800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff965984c46000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff965984c46000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.7343 s, 7.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.95896 s, 26.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.15195 s, 33.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.79475 s, 18.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.62891 s, 12.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15323 s, 25.2 MB/s PASS 16d (58s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:05:54 (1785391554) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.435883 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903338 s, 11.6 MB/s PASS 16e (10s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:06:04 (1785391564) TIMEOUT PASS 16f (32s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:06:36 (1785391596) TIMEOUT PASS 16g (32s) == sanityn test 16h: mmap read after truncate file ======= 02:07:08 (1785391628) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 02:07:17 (1785391637) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 02:07:27 (1785391647) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.13702 s, 6.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.2159 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.6937 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.30577 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.78516 s, 6.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36341 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.94932 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1179 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.55773 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.55241 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.89877 s, 5.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.70709 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.89567 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.70834 s, 5.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.09981 s, 5.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.90095 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.50494 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.52623 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.74562 s, 5.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (58s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:08:24 (1785391704) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Jul 30 02:09:27 EDT 2026 + fsx 2 done: rc=0 Thu Jul 30 02:09:27 EDT 2026 + fsx 3 done: rc=0 Thu Jul 30 02:09:27 EDT 2026 + fsx 4 done: rc=0 Thu Jul 30 02:09:27 EDT 2026 + fsx 5 done: rc=0 Thu Jul 30 02:09:27 EDT 2026 PASS 16k (77s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:09:42 (1785391782) fail_loc=0x8000030a PASS 17 (14s) == sanityn test 18: mmap sanity check =========================================================================================== 02:09:56 (1785391796) mmap test1: basic mmap operation (PASS, 0.24723s) mmap test2: MAP_PRIVATE not write back (PASS, 0.12252s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.217s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 5.1083s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.31995s) mmap test6: check mmap write/read content on two nodes (PASS, 0.5136s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 3.2812s) mmap test11: multi-thread shared mmap (PASS, 4.8489s) mmap test10: mtime not change for readonly mmap access (PASS, 1.2066s) PASS 18 (38s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:10:34 (1785391834) oleg232-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:10:38 (1785391838) PASS 20 (11s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:10:49 (1785391849) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 02:10:56 (1785391856) now is 1785391859 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 new atime is 1785391920 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:12:06 (1785391926) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:12:14 (1785391934) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:12:23 (1785391943) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:12:32 (1785391952) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (5s) == sanityn test 26a: allow mtime to get older ============ 02:12:37 (1785391957) utime: good mknod times 1785391959 <= 1785391960 <= 1785391960 for /mnt/lustre/f26a utime: good mknod times 1785391959 <= 1785391960 <= 1785391960 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (12s) == sanityn test 26b: sync mtime between ost and mds ====== 02:12:49 (1785391969) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:13:00 (1785391980) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502397 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 02:13:02.000000000 -0400 Birth: 2026-07-30 02:13:02.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502397 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-07-30 02:13:02.000000000 -0400 Birth: 2026-07-30 02:13:02.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:13:13 (1785391993) 0 PASS 30 (12s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:13:25 (1785392005) fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:13:35 (1785392015) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.132@tcp:/lustre 7542784 40960 7496704 1% /mnt/lustre2 PASS 31b (27s) == sanityn test 31r: open-rename(replace) race =========== 02:14:02 (1785392042) -rw-r--r-- 1 root root 0 Jul 30 02:14 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (12s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:14:14 (1785392054) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43157 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:14:25 (1785392065) PASS 31t (12s) == sanityn test 32b: lockless i/o ======================== 02:14:37 (1785392077) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (6s) == sanityn test 32c: contention detection ================ 02:14:43 (1785392083) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.86699 s, 28.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40478 s, 37.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.44524 s, 15.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.58231 s, 14.6 MB/s contention_seconds 4, contention_events 22 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47707 s, 35.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71843 s, 30.5 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51945 s, 34.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53517 s, 34.2 MB/s PASS 32c (35s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:15:20 (1785392120) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:15:25 (1785392125) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (6s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:15:31 (1785392131) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (6s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:15:37 (1785392137) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (6s) == sanityn test 34: no lock timeout under IO ============= 02:15:43 (1785392143) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid in IDLE state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid in FULL state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid in IDLE state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid in FULL state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff965984c46000.ost_server_uuid in IDLE state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff965984c46000.ost_server_uuid in FULL state after 0 sec PASS 34 (169s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:18:32 (1785392312) fail_loc=0x80000317 Race attempt 0 Wait for 50464 50486 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50464 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.132@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.132@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 - open/close 461 (time 1785392332.36 total 10.03 last 45.98) - open/close 872 (time 1785392342.39 total 20.06 last 40.96) - open/close 1444 (time 1785392352.39 total 30.06 last 57.20) - open/close 2075 (time 1785392362.40 total 40.06 last 63.09) - open/close 2800 (time 1785392372.41 total 50.07 last 72.41) - open/close 3194 (time 1785392382.48 total 60.15 last 39.12) - open/close 3644 (time 1785392392.49 total 70.16 last 44.95) total: 4000 open/close in 76.83 seconds: 52.07 ops/second PASS 35 (87s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:19:59 (1785392399) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07269 s, 48.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.761764 s, 68.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7485440) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.811251 s, 64.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.585093 s, 89.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7419904) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.919222 s, 57.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.999415 s, 52.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.770803 s, 68.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7424000) after(7475200) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.996278 s, 52.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7475200) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.70131 s, 74.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7489536) after_dd(7426048) after(7477248) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.719188 s, 72.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7477248) after_dd(7426048) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01521 s, 51.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7433216) after(7496704) PASS 36 (330s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:25:30 (1785392730) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 - create 974 (time 1785392744.33 total 10.00 last 97.37) - create 2595 (time 1785392754.33 total 20.01 last 162.06) - create 4040 (time 1785392764.33 total 30.01 last 144.49) - create 5082 (time 1785392774.33 total 40.01 last 104.19) - create 6118 (time 1785392784.34 total 50.01 last 103.54) - create 6785 (time 1785392794.40 total 60.07 last 66.30) - create 7634 (time 1785392804.40 total 70.07 last 84.88) - create 8451 (time 1785392814.40 total 80.08 last 81.67) - create 9288 (time 1785392824.40 total 90.08 last 83.68) total: 10000 create in 97.99 seconds: 102.05 ops/second PASS 37 (117s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:27:27 (1785392847) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:27:35 (1785392855) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:27:44 (1785392864) repeat after cancel_lru_locks PASS 39c (10s) == sanityn test 39d: sync write should update mtime ====== 02:27:55 (1785392875) fail_loc=0x411 fail_loc=0 PASS 39d (12s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:28:09 (1785392889) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (24s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:28:33 (1785392913) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (22s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:28:55 (1785392935) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:29:15 (1785392955) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (21s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:29:37 (1785392977) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (17s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:29:54 (1785392994) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:30:10 (1785393010) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (20s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:30:30 (1785393030) Conflict PASS 41d (17s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:30:47 (1785393047) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:31:02 (1785393062) Conflict PASS 41f (16s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:31:18 (1785393078) Conflict PASS 41g (17s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:31:35 (1785393095) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 02:31:50 (1785393110) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1244s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:52:34 (1785394354) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (20s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:52:54 (1785394374) open: File exists Conflict PASS 42b (18s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:53:12 (1785394392) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:53:27 (1785394407) Conflict PASS 42d (18s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:53:45 (1785394425) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (17s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:54:02 (1785394442) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:54:17 (1785394457) Conflict PASS 42g (17s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:54:34 (1785394474) Conflict PASS 42h (19s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:54:53 (1785394493) PASS 43a (68s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:56:01 (1785394561) Conflict PASS 43b (16s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:56:17 (1785394577) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:56:34 (1785394594) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (16s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:56:50 (1785394610) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:57:06 (1785394626) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:57:21 (1785394641) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:57:37 (1785394657) Conflict PASS 43h (17s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:57:54 (1785394674) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:57:59 (1785394679) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (144s) == sanityn test 43k: unlink vs create ==================== 03:00:23 (1785394823) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1508s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:25:31 (1785396331) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jul 30 03:25:42 EDT 2026 PASS 44a (17s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:25:48 (1785396348) open: File exists Conflict PASS 44b (17s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:26:06 (1785396366) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (18s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:26:23 (1785396383) Conflict PASS 44d (16s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:26:39 (1785396399) Conflict PASS 44e (21s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:27:00 (1785396420) Conflict PASS 44f (18s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:27:18 (1785396438) Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:27:34 (1785396454) Conflict PASS 44h (16s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:27:50 (1785396470) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:27:54 (1785396474) PASS 45a (156s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:30:30 (1785396630) Conflict PASS 45b (18s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:30:48 (1785396648) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:31:04 (1785396664) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:31:21 (1785396681) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:31:38 (1785396698) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (17s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:31:55 (1785396715) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:32:11 (1785396731) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:32:25 (1785396745) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 03:32:30 (1785396750) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1533s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:58:03 (1785398283) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (16s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:58:19 (1785398299) open: File exists Conflict PASS 46b (18s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:58:37 (1785398317) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:58:53 (1785398333) Conflict PASS 46d (17s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:59:09 (1785398349) Conflict PASS 46e (19s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:59:28 (1785398368) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:59:42 (1785398382) Conflict PASS 46g (18s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 04:00:00 (1785398400) Conflict PASS 46h (18s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 04:00:18 (1785398418) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 04:00:23 (1785398423) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 04:00:26 (1785398426) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 04:00:31 (1785398431) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 04:00:36 (1785398436) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 04:00:40 (1785398440) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 04:00:45 (1785398445) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 04:00:50 (1785398450) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 04:00:54 (1785398454) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0881789 s, 116 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 04:01:08 (1785398468) 0+1 records in 0+1 records out 158 bytes copied, 0.0640756 s, 2.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218487) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 04:01:19 (1785398479) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0323077 s, 31.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181207 s, 5.8 MB/s 3145728 PASS 51b (23s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 04:01:42 (1785398502) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0349892 s, 29.3 kB/s PASS 51c (15s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 04:01:57 (1785398517) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055659 s, 18.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (10s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 04:02:08 (1785398528) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 04:02:17 (1785398537) llite.lustre-ffff965984c46000.xattr_cache=1 llite.lustre-ffff965991e9f800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:19.000000000 -0400 Modify: 2026-07-30 04:02:19.000000000 -0400 Change: 2026-07-30 04:02:19.000000000 -0400 Birth: 2026-07-30 04:02:19.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:25.000000000 -0400 Modify: 2026-07-30 04:02:26.000000000 -0400 Change: 2026-07-30 04:02:26.000000000 -0400 Birth: 2026-07-30 04:02:26.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:32.000000000 -0400 Modify: 2026-07-30 04:02:32.000000000 -0400 Change: 2026-07-30 04:02:32.000000000 -0400 Birth: 2026-07-30 04:02:32.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:32.000000000 -0400 Modify: 2026-07-30 04:02:33.000000000 -0400 Change: 2026-07-30 04:02:33.000000000 -0400 Birth: 2026-07-30 04:02:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:38.000000000 -0400 Modify: 2026-07-30 04:02:39.000000000 -0400 Change: 2026-07-30 04:02:39.000000000 -0400 Birth: 2026-07-30 04:02:38.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 04:02:38.000000000 -0400 Modify: 2026-07-30 04:02:39.000000000 -0400 Change: 2026-07-30 04:02:39.000000000 -0400 Birth: 2026-07-30 04:02:38.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 04:02:51 (1785398571) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 04:03:08 (1785398588) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (19s) == sanityn test 55c: rename vs unlink orphan target dir == 04:03:27 (1785398607) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (23s) == sanityn test 55d: rename file vs link ================= 04:03:50 (1785398630) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (17s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 04:04:08 (1785398648) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (5s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 04:04:13 (1785398653) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (25s) == sanityn test 55g: rename: race with trylock =========== 04:04:38 (1785398678) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (25s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 04:05:03 (1785398703) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1785398705 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.445541 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (146s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 04:07:29 (1785398849) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1785398861 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.0803 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (133s) == sanityn test 60: Verify data_version behaviour ======== 04:09:43 (1785398983) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 04:09:53 (1785398993) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 04:10:02 (1785399002) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 04:10:12 (1785399012) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.037705 s, 1.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 04:10:17 (1785399017) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0234032 s, 2.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 04:10:22 (1785399022) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (8s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 04:10:30 (1785399030) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0494279 s, 1.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (6s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 04:10:37 (1785399037) llite.lustre-ffff965984c46000.xattr_cache=1 llite.lustre-ffff965991e9f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (10s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 04:10:47 (1785399047) llite.lustre-ffff965984c46000.xattr_cache=1 llite.lustre-ffff965991e9f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (11s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 04:10:59 (1785399059) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0538423 s, 19.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0148221 s, 69.1 kB/s 231314: get lock2 231315: get lock1 231314: done 231314: get lock1 231315: done 231315: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231315: cannot get lock2: Resource deadlock avoided 231315: exit rc=0 231314: done 231314: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 04:11:13 (1785399073) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0431113 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032659 s, 32.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231897 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (15s) == sanityn test 76: Verify MDT open_files listing ======== 04:11:29 (1785399089) opening files: ........ closing files: ........ PASS 76 (99s) == sanityn test 77a: check FIFO NRS policy =============== 04:13:09 (1785399189) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.558658 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379424 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753825 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718959 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140982 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154576 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183771 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315318 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209994 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357183 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205099 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342908 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213051 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278273 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266035 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425911 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.652567 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134051 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428583 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154178 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176873 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10843 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919341 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133376 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183471 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172262 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188366 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218372 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257131 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232279 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241309 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266051 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29686 s, 3.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (18s) == sanityn test 77b: check CRR-N NRS policy ============== 04:13:27 (1785399207) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313099 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532297 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599214 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110175 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183418 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203058 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19744 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132897 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212785 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210679 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151663 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245232 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153634 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229778 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202344 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257708 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319662 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467269 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226382 s, 463 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00184069 s, 570 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900188 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800269 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151658 s, 6.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650248 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643831 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134325 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674252 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636752 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583267 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562316 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869808 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519291 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476752 s, 22.0 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.346837 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472207 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601918 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794633 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056771 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177992 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11252 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121977 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21723 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247134 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229202 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28263 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261861 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319207 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254811 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24931 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318532 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941827 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101244 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995501 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171145 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104116 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107201 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125293 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12663 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134212 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142251 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163441 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175707 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14755 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183249 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190852 s, 5.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (17s) == sanityn test 77c: check ORR NRS policy ================ 04:13:44 (1785399224) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392837 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383188 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318788 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244971 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108168 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235916 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713677 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104814 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704679 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061556 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894796 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727252 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780799 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562051 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22021 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147021 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143602 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896272 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976359 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103639 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491395 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120329 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913109 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0087292 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134734 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637929 s, 164 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145461 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043724 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131308 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174054 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106068 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852072 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320333 s, 327 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.325548 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395443 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853206 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092178 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150429 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149729 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992159 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147474 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183823 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777244 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119992 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164827 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186958 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183001 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144249 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196498 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282601 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103321 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733533 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234679 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102314 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0021682 s, 484 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936387 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930674 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673071 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698363 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100292 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00774675 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00646242 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996163 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117561 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115891 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676763 s, 155 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444225 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547844 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101436 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115649 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116539 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133049 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150714 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149619 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117411 s, 8.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223775 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206715 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170397 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204611 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242752 s, 4.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246984 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230231 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288428 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799866 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570101 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171118 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346866 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947868 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692631 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343658 s, 305 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870041 s, 12.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00211009 s, 497 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830795 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551821 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162362 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124808 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117186 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433674 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698835 s, 15.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 04:14:04 (1785399244) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.452115 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137766 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145573 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889368 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340825 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221329 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215359 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185995 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257897 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224942 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318395 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432252 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269899 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415501 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389121 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419337 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435887 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450376 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471251 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874752 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153318 s, 68.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017303 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130422 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25081 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190893 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140353 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063127 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171951 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291896 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205972 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064375 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052402 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525664 s, 19.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.594913 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647091 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732246 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302461 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027808 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285058 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238661 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152705 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088072 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262895 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272959 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35636 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254195 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408104 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199936 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272097 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30714 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099714 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863823 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017352 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808583 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012899 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116703 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124832 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00212567 s, 493 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140018 s, 74.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846685 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104194 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00875199 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271215 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137435 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132132 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187501 s, 55.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301704 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190445 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337914 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882749 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122147 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152733 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16956 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147764 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540834 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219679 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140462 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231227 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171934 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202587 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225374 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222492 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289181 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266208 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800963 s, 131 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20962 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997454 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161925 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706334 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638018 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547366 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437566 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395416 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377495 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350329 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208312 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231485 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307803 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032132 s, 32.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (25s) == sanityn test 77e: check TBF NID nrs policy ============ 04:14:30 (1785399270) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.32@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154875 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206347 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291686 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575836 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583546 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661774 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838556 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770387 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859968 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883185 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719027 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884701 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949085 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978021 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908248 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106007 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103552 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499269 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544235 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659612 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446114 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493028 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619668 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306572 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04543 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058143 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557849 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698102 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538226 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011578 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902651 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720068 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891487 s, 11.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.454157 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482448 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020788 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162522 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991217 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715681 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469191 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241043 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117305 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159306 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148354 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262885 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250914 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36033 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263761 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278341 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30861 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750569 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118359 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00198674 s, 528 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592765 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239172 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689818 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827975 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095697 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985626 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303226 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958467 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10199 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112264 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130989 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12751 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118782 s, 8.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40635 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360591 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196871 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825237 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109268 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429965 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162977 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133187 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163051 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270279 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275485 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270914 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365139 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308552 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351847 s, 3.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352467 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473566 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184389 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170527 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153475 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051891 s, 20.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167396 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320237 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705963 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804663 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.514252 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00200357 s, 523 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288192 s, 3.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063337 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575823 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411461 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228052 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337043 s, 31.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180732 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246151 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697185 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993113 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991959 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853518 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158339 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155737 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121787 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115634 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249964 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183971 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173236 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192156 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269881 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251726 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230691 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290112 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293932 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899027 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031225 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227929 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360952 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350858 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308058 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374021 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489096 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049707 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425081 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050951 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512304 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599436 s, 17.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (27s) == sanityn test 77f: check TBF JobID nrs policy ========== 04:14:56 (1785399296) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3193 s, 52.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428672 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294611 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511992 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697801 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760599 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590157 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126436 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204985 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109873 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249443 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23591 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116187 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165011 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179202 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218808 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286464 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624907 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895107 s, 117 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108497 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00519203 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843331 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666625 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939065 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00192313 s, 545 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0083 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094858 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562949 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590536 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237278 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084868 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647115 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653251 s, 16.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.466524 s, 36.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203045 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601864 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476519 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453935 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119073 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123126 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172252 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196461 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224318 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132124 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236968 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360783 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295199 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36752 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288979 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306059 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00200624 s, 523 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14046 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125891 s, 8.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387241 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140685 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101713 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620195 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00229836 s, 456 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228945 s, 458 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214573 s, 489 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245846 s, 427 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747719 s, 14.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010251 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211974 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285389 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470107 s, 22.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.360528 s, 46.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470215 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104543 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146148 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123549 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155826 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181353 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109681 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252377 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199117 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269121 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167818 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282125 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171578 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326151 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344999 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408894 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745213 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893373 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834598 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00840991 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900652 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119247 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105387 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119265 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991604 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114694 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109846 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885523 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121535 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124675 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16935 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954537 s, 11.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412899 s, 40.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494392 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917557 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190812 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175232 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243209 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227507 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193461 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255274 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268778 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253367 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227109 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315251 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339047 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270014 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238097 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372557 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103528 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499871 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220987 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147412 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203091 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205503 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108195 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193079 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525759 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611061 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474664 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137198 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048971 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22629 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041374 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252888 s, 41.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (28s) == sanityn test 77g: Change TBF type directly ============ 04:15:24 (1785399324) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 04:15:35 (1785399335) oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 PASS 77h (16s) == sanityn test 77i: Change rank of TBF rule ============= 04:15:51 (1785399351) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (40s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 04:16:31 (1785399391) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291262 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398273 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728527 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861694 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127965 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485198 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551069 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11541 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110207 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116314 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153431 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171322 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119892 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193086 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125444 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199069 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23499 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00196795 s, 533 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701184 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856613 s, 122 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0021466 s, 488 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024236 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124173 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00235883 s, 445 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112555 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374536 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142596 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108048 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16898 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171002 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25785 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138807 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897701 s, 11.7 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.11957 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7381 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (70s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 04:17:41 (1785399461) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25808 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102435 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120065 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118421 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113708 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809414 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222715 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211158 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233401 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221383 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318717 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282919 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267532 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253507 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247562 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250959 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335815 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976116 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114552 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107324 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100884 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120778 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12557 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108327 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139232 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976638 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107898 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145963 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106616 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114693 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159486 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117954 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143068 s, 7.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.8057 s, 5.0 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.228 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30073 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146227 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141965 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184585 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189317 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212767 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346268 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273116 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256478 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341111 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384809 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372053 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356578 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403558 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366951 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482332 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486905 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316532 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111493 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633804 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108554 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106842 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181176 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176219 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215722 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515856 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0023142 s, 453 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167706 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181981 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191548 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519341 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190647 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049175 s, 21.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.5281 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.905 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (169s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 04:20:31 (1785399631) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.50923 s, 11.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11117 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111117 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105624 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105959 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166307 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183396 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203123 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215754 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175009 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236273 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242071 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25778 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257205 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320243 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284709 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394075 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085492 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100634 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13417 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124985 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152288 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1317 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166115 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16492 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167074 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137584 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154897 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165033 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156133 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173695 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172412 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19884 s, 5.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.4886 s, 4.9 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0772 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29681 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090015 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510893 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106348 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144135 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092342 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152643 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178811 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154613 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14451 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214908 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295649 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259991 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281126 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305835 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298037 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342231 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123067 s, 8.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129832 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133845 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131517 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824389 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156409 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159173 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160631 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141557 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176578 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151839 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130354 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182018 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202522 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225279 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186569 s, 5.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.8666 s, 4.8 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9447 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (168s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 04:23:19 (1785399799) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.418274 s, 40.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427402 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014312 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562485 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124215 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113987 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955517 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134398 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138554 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147256 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135085 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130789 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181332 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149198 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183732 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191542 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253432 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315838 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378269 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058773 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494871 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670359 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441993 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432498 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399973 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346171 s, 30.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288906 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766565 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552935 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457612 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578411 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467243 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650698 s, 16.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9572 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0322 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.588919 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509033 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368816 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711722 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063702 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316307 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150188 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144391 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863424 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181241 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199256 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143914 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142893 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216086 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19265 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253049 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301463 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323745 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878374 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424052 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518367 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585841 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428732 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102648 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587735 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643563 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698973 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722668 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697119 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793144 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869677 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934475 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873199 s, 12.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.6369 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2854 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.782641 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620179 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168988 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991402 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18337 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212343 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229368 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102712 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240354 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188633 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219744 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225188 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333947 s, 3.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302339 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31807 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351754 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419115 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834797 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826932 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097235 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173781 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172648 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224173 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995116 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134616 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140346 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127237 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216337 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220371 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154739 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284875 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17248 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206252 s, 5.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3941 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26052 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.604693 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809927 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951526 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121218 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166361 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123957 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164522 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11699 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187321 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211492 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140525 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193226 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237962 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411117 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377963 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415016 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413075 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051239 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436584 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028887 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514369 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357102 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453403 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385255 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540395 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461747 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731642 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13343 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862929 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1481 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802445 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858704 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110701 s, 9.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9275 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.917 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7387 s, 8.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5302 s, 10.0 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.12803 s, 14.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657999 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408152 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119926 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194828 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924789 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112504 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132368 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201572 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263019 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252983 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369889 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331315 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362576 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41486 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459953 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481167 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117102 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025338 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219639 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127666 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156264 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280147 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323471 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129934 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31595 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305244 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242609 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874936 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904117 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26649 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298701 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117216 s, 8.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.5821 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7739 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.494514 s, 33.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332578 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380134 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738055 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115345 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425616 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783435 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13644 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566688 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649659 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108405 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281865 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230358 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211411 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234996 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286646 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241818 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056517 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785271 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295571 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621559 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383422 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956542 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112498 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540117 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634994 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669599 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100686 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102218 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113563 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890714 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979725 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130529 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.2947 s, 6.4 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3207 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.745799 s, 22.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774513 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812458 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128894 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117732 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192819 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182991 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157246 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197239 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180524 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229912 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200918 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243231 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309719 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307359 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290228 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275892 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754084 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107619 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061813 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797616 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512429 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984786 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389638 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11451 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650469 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148806 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137463 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983215 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912216 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111539 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103973 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124379 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.2757 s, 7.3 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0398 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.7473 s, 7.1 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.16 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (509s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 04:31:48 (1785400308) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.754036 s, 22.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401001 s, 26.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593433 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476422 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814079 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210386 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176098 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190306 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150879 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234882 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260966 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173013 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263954 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247426 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287621 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220524 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295218 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798349 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639461 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602642 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156503 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00201604 s, 520 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237796 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107995 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1035 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00658828 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903957 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830235 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180587 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678898 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353422 s, 29.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599347 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533985 s, 19.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8568 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2274 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (63s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 04:32:51 (1785400371) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.891017 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121113 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109647 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289281 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309942 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330473 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258076 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337202 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21446 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305694 s, 3.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302601 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252055 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285471 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419874 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359056 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44553 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455343 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279533 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253249 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787226 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324864 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122662 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486933 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479581 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493756 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105317 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602304 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664508 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761593 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756295 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843553 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942793 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106044 s, 9.9 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.8239 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4084 s, 10.1 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (64s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 04:33:55 (1785400435) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619203 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06661 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435877 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419266 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712139 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253239 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22727 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259083 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266193 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283284 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233981 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242523 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225543 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310816 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259615 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323483 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331696 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636074 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893291 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698162 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260723 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765316 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528968 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408177 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609076 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257407 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681823 s, 15.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686619 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552889 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738692 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169792 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618225 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480937 s, 21.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.9236 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.84451 s, 15.3 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:34:43 (1785400483) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.655657 s, 25.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510745 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647479 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486741 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978165 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108766 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191601 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135229 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200845 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251281 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25065 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21881 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289452 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22249 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22265 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418183 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408363 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340526 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100956 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518384 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795598 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871965 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875047 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832832 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912119 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793394 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106295 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111906 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10322 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139446 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321253 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116015 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323243 s, 32.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.5644 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1029 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.2754 s, 7.3 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97204 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (117s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:36:40 (1785400600) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0084 s, 16.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133417 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190672 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184696 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106288 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196944 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140782 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24172 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17526 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189707 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344663 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290645 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304978 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296024 s, 3.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382878 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284174 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467124 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576131 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838343 s, 12.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276971 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445629 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532248 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837858 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817895 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821786 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109241 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109836 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915216 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788769 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108731 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749088 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760909 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404809 s, 259 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1728 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4578 s, 5.1 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (77s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:37:57 (1785400677) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.870121 s, 19.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772083 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073436 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318113 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217218 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121376 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814341 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25506 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151701 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255344 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285008 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168602 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289399 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185485 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225391 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221542 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34276 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965991e9f800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff965991e9f800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720608 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227913 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310863 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412955 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396712 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112126 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121934 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630375 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656781 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691098 s, 15.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748683 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797822 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796385 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789691 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824525 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108623 s, 9.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.482 s, 7.2 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2476 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.707096 s, 23.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368616 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759685 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126545 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146297 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209583 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255757 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218346 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263873 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26052 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217105 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209722 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19177 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239782 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351728 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365067 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293796 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533218 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629477 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120026 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706656 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793154 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102641 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569549 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766596 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670627 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773552 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992669 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829582 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108469 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856283 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913763 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152565 s, 6.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.4311 s, 7.3 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1887 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.7832 s, 7.1 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0486 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (172s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 04:40:49 (1785400849) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre opts: Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg232-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg232-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg232-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.32,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/66907/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.32,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg232-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Started clients oleg232-client.virtnet: 192.168.202.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Started clients oleg232-client.virtnet: 192.168.202.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff965989c6b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff965991a6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff965989c6b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff965991a6c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.76635 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151499 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296665 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371929 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16988 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12518 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420443 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500442 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312185 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35704 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51341 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517995 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301657 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396465 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262975 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.623153 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343808 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482964 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443279 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455143 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463937 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466518 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509253 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527588 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521144 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525486 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524873 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.534394 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531227 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529159 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521042 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532229 s, 2.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.552667 s, 1.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.0175 s, 7.0 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.69975 s, 15.7 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.908281 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14281 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171769 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311182 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29794 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345147 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351262 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471078 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462441 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485985 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504195 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.542265 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.610238 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571938 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.660281 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.655831 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.792465 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102808 s, 10.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770233 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111692 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115118 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944273 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131864 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114444 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174233 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226142 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234293 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179905 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158311 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24762 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123702 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154065 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186461 s, 5.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 18.4001 s, 5.7 MB/s Write runtime is 19 s, speed is 5.263157 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3455 s, 10.1 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (298s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 04:45:48 (1785401148) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (24s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 04:46:12 (1785401172) On MGS 192.168.202.132, active = nodemap.active=1 On MGS 192.168.202.132, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.880362 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126629 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106956 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12163 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192675 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416574 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251147 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382376 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402035 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41872 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37297 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37067 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503591 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.561213 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431427 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510883 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475583 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377281 s, 2.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376815 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417256 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381117 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323025 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342778 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38167 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344597 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344282 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362443 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384141 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357455 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388273 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396504 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437018 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405056 s, 2.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5754 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.27169 s, 16.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.132, TBF.id = On MGS 192.168.202.132, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.887662 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558851 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618045 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104983 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508177 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215678 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228706 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351345 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316003 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288175 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22504 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424189 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264423 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104966 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480354 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401224 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.699594 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345343 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060692 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117512 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12648 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629014 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500463 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538045 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771125 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880599 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165604 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702937 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198028 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211642 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832289 s, 12.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880815 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236089 s, 4.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.982 s, 5.8 MB/s Write runtime is 19 s, speed is 5.263157 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.44079 s, 23.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 On MGS 192.168.202.132, active = nodemap.active=1 On MGS 192.168.202.132, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.623175 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841409 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101802 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178393 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193033 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187157 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240997 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255793 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268581 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199335 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299323 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246257 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306204 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271947 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339517 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353164 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315553 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068974 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614747 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844592 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121795 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220821 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214176 s, 490 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449188 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101016 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00895668 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00212724 s, 493 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116679 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908418 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00590064 s, 178 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113586 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168505 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982538 s, 10.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.311 s, 7.9 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.326 s, 10.2 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.132, TBF.id = On MGS 192.168.202.132, active = nodemap.active=0 PASS 77kj (238s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:50:10 (1785401410) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (13s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:50:24 (1785401424) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg232-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.05199 s, 259 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg232-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0506 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (64s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:51:27 (1785401487) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.726162 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11054 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612232 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10686 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116885 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136711 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519978 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157322 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183884 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107858 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274936 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233619 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264844 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279236 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153259 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff965989c6b000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31305 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379822 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337009 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239816 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252942 s, 41.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982577 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423806 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332599 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521232 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490718 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545701 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500714 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456084 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500622 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987325 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff965989c6b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750475 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641876 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072491 s, 14.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1333 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03558 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.810802 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970135 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516721 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304016 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381786 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169911 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146421 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284113 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139776 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107444 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290417 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309007 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323369 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407143 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180935 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270505 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127894 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175374 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223073 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210125 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236444 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187305 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119903 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1156 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162028 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256555 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20743 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166262 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227282 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253578 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245283 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302165 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316712 s, 3.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg232-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 28.0044 s, 3.7 MB/s Write runtime is 29 s, speed is 3.448275 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.9055 s, 15.2 MB/s Read runtime is 8 s, speed is 12.500000 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (132s) == sanityn test 77o: Changing rank should not panic ====== 04:53:39 (1785401619) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (17s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:53:56 (1785401636) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (146s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:56:22 (1785401782) mds.MDS.mdt.nrs_policies=tbf oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg232-client: oleg232-server: ssh exited with exit code 36 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (56s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:57:19 (1785401839) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272413 kill md_thread xargs PASS 77r (67s) == sanityn test 77s: Check TBF LRU shrinker ============== 04:58:26 (1785401906) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (97s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 05:00:02 (1785402002) oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg232-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (12s) == sanityn test 79: xattr: intent error ================== 05:00:14 (1785402014) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935567 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:00:17.000000000 -0400 Modify: 2026-07-30 05:00:17.000000000 -0400 Change: 2026-07-30 05:00:17.000000000 -0400 Birth: 2026-07-30 05:00:17.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (22s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 05:00:36 (1785402036) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 05:00:40 (1785402040) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (6s) == sanityn test 81a: rename and stat under striped directory ========================================================== 05:00:46 (1785402046) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (5s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 05:00:51 (1785402051) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (6s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 05:00:57 (1785402057) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 05:01:02 (1785402062) - open/close 548 (time 1785402080.36 total 10.02 last 54.68) - open/close 1102 (time 1785402090.36 total 20.02 last 55.39) - open/close 1571 (time 1785402100.36 total 30.03 last 46.87) - open/close 2018 (time 1785402110.38 total 40.05 last 44.63) - open/close 2409 (time 1785402120.39 total 50.05 last 39.07) total: 2500 open/close in 52.09 seconds: 48.00 ops/second snapshot_time 1785402250.181454167 secs.nsecs start_time 1785402066.548015051 secs.nsecs elapsed_time 183.633439116 secs.nsecs open 2505 samples [usecs] 155 28625 4328896 18904145312 close 2504 samples [usecs] 135 8324 777454 484157892 mknod 2501 samples [usecs] 445 28521 3768222 15409914956 unlink 2500 samples [usecs] 675 94405 5670300 54154399426 mkdir 2 samples [usecs] 1650 6001 7651 38734501 rmdir 2 samples [usecs] 10747 49327 60074 2548650938 rename 2500 samples [usecs] 413 128638 23702446 536144190824 getattr 3761 samples [usecs] 9 27691 319869 1524874607 parallel_rename_file 2500 samples [usecs] 413 128638 23702446 536144190824 crossdir_rename 2500 samples [usecs] 413 128638 23702446 536144190824 PASS 81d (194s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 05:04:16 (1785402256) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7613 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7613 PASS 82 (11s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 05:04:27 (1785402267) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 05:04:31 (1785402271) fail_loc=0x8000060b PASS 84 (19s) == sanityn test 85: Lustre API root cache race =========== 05:04:50 (1785402290) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1785402292 Finishing test test1 at 1785402299 PASS 85 (15s) == sanityn test 90: open/create and unlink striped directory ========================================================== 05:05:05 (1785402305) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 05:05:09 (1785402309) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (5s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:05:14 (1785402314) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:05:18 (1785402318) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5694 0x163e 0x240000400 1 5655 0x1617 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5656 0x1618 0x280000400 0 5695 0x163f 0x240000400 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 05:05:38 (1785402338) osc.lustre-OST0000-osc-ffff965989c6b000.idle_timeout=0 osc.lustre-OST0000-osc-ffff965991a6c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff965989c6b000.idle_timeout=0 osc.lustre-OST0001-osc-ffff965991a6c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.416379 s, 19.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00581 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 288719 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.149474 s, 27.4 kB/s osc.lustre-OST0000-osc-ffff965989c6b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff965991a6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff965989c6b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff965991a6c800.idle_timeout=debug PASS 94 (23s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 05:06:02 (1785402362) llite.lustre-ffff965989c6b000.fast_read=0 llite.lustre-ffff965991a6c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151528 s, 13.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff965989c6b000.fast_read=1 llite.lustre-ffff965991a6c800.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:06:24 (1785402384) llite.lustre-ffff965989c6b000.fast_read=0 llite.lustre-ffff965991a6c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00300674 s, 4.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff965989c6b000.fast_read=1 llite.lustre-ffff965991a6c800.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:06:39 (1785402399) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:06:44 (1785402404) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00576573 s, 710 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (11s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:06:54 (1785402414) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203651 s, 10.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:07:03 (1785402423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1876 s, 11.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 05:07:12 (1785402432) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0120785 s, 993 kB/s PASS 100e (12s) == sanityn test 101a: Discard DoM data on unlink ========= 05:07:23 (1785402443) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0200203 s, 205 kB/s PASS 101a (12s) == sanityn test 101b: Discard DoM data on rename ========= 05:07:35 (1785402455) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0233423 s, 175 kB/s PASS 101b (10s) == sanityn test 101c: Discard DoM data on close-unlink === 05:07:45 (1785402465) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0198551 s, 206 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 05:07:55 (1785402475) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 f0 0e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xef0:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 05:08:04 (1785402484) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1785402487 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1785402491 Finishing test test23 at 1785402491 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:08:20 (1785402500) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 05:08:25 (1785402505) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 05:08:54 (1785402534) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (5s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:08:59 (1785402539) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00934209 s, 110 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938103 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:09:01.000000000 -0400 Modify: 2026-07-30 05:09:01.000000000 -0400 Change: 2026-07-30 05:09:01.000000000 -0400 Birth: 2026-07-30 05:09:01.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938103 -rw-r--r-- 1 0 0 1785402541 1785402541 1785402541 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938103 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:09:01.000000000 -0400 Modify: 2026-07-30 05:09:01.000000000 -0400 Change: 2026-07-30 05:09:01.000000000 -0400 Birth: 2026-07-30 05:09:01.000000000 -0400 1785402541 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 05:09:07 (1785402547) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 05:09:16 (1785402556) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.394458 s, 26.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5663 0x161f 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:09:28 (1785402568) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.436474 s, 24.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5704 0x1648 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7613 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (19s) == sanityn test 108a: lseek: parallel updates ============ 05:09:48 (1785402588) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0156358 s, 640 kB/s llite.lustre-ffff965989c6b000.hybrid_io=0 llite.lustre-ffff965991a6c800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 5.50056 s, 1.5 MB/s PASS 108a (18s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:10:06 (1785402606) 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 PASS 109 (70s) == sanityn test 110: do not grant another lock on resend ========================================================== 05:11:17 (1785402677) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 05:11:19.000000000 -0400 Modify: 2026-07-30 05:11:19.000000000 -0400 Change: 2026-07-30 05:11:19.000000000 -0400 Birth: 2026-07-30 05:11:19.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg232-client.virtnet oleg232-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 05:12:41 (1785402761) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 05:12:45 (1785402765) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (5s) == sanityn test 113: check servers of specified fs ======= 05:12:51 (1785402771) lustre-OST0000-osc-ffff965983c31800 active. lustre-OST0001-osc-ffff965983c31800 active. lustre-OST0000-osc-ffff965983c31800 active. lustre-OST0001-osc-ffff965983c31800 active. MGC192.168.202.132@tcp active. PASS 113 (14s) == sanityn test 114: implicit default LMV inherit ======== 05:13:06 (1785402786) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 05:13:11 (1785402791) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 05:13:16 (1785402796) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (6s) == sanityn test 121: trunc append race =================== 05:13:22 (1785402802) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028214 s, 726 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.153888 s, 133 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (13s) == sanityn test 200: service remains healthy while able to process request ========================================================== 05:13:35 (1785402815) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1387 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1275 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.18 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2027 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2331 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.197 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.294 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1722 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4467 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4637 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4868 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5918 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5651 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.5338 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3492 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6599 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.668 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6966 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9659832ef800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9659832ef800.ost_server_uuid in IDLE state after 0 sec PASS 200 (93s) cleanup: ====================================================== == sanityn test complete, duration 12064 sec ============= 05:15:10 (1785402910) === sanityn: start cleanup 05:15:12 (1785402912) === Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 05:19:49 (1785403189) ===