-----============= acceptance-small: sanityn ============----- Thu Jan 1 19:50:30 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg317-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 19:50:48 (1767315048) === Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Mount client oleg317-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg317-client.virtnet: executing check_config_client /mnt/lustre oleg317-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg317-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d57c2bba800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d57d015c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57c2bba800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57d015c000.idle_timeout=debug disable quota as required oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:51:39 (1767315099) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165277 s, 63.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f9350] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:51:42 (1767315102) PASS 0 (12s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:51:54 (1767315114) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:52:02 (1767315122) -rw-r--r-- 1 root root 0 Jan 1 19:52 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:52:08 (1767315128) -rw-r--r-- 1 root root 0 Jan 1 19:52 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:52:14 (1767315134) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:52:20 (1767315140) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:52:27 (1767315147) total 1 -rwxrwxrwx 1 root root 0 Jan 1 19:52 f2a -rwxrwxrwx 1 root root 0 Jan 1 19:52 f2b total 1 -rwxrwxrwx 1 root root 0 Jan 1 19:52 f2a -rwxrwxrwx 1 root root 0 Jan 1 19:52 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0223436 s, 22.9 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:52:33 (1767315153) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 19:52:36 (1767315156) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.463755 s, 4.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:52:42 (1767315162) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:52:48 (1767315168) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:52:55 (1767315175) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 19:53:00 (1767315180) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:53:06 (1767315186) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:53:12 (1767315192) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:53:18 (1767315198) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 19:53:25 (1767315205) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344746 s, 30.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0390165 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:53:31 (1767315211) 1+0 records in 1+0 records out 1 byte copied, 0.0114171 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0313093 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0125471 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00980239 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0260115 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00742283 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00890306 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121073 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00917645 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00505139 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00839658 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110565 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:53:38 (1767315218) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0165592 s, 186 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0289349 s, 142 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000279445 s, 11.0 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:53:43 (1767315223) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:53:50 (1767315230) warning: '-runtime' deprecated, use '-t runtime' instead - link 353 (time 1767315242.44 total 10.01 last 35.25) - link 707 (time 1767315252.47 total 20.04 last 35.30) - link 1101 (time 1767315262.47 total 30.05 last 39.39) - link 1372 (time 1767315272.48 total 40.05 last 27.08) - link 1631 (time 1767315282.48 total 50.05 last 25.90) - link 1963 (time 1767315292.51 total 60.08 last 33.11) - link 2298 (time 1767315302.51 total 70.08 last 33.50) - link 2665 (time 1767315312.52 total 80.09 last 36.66) - link 3020 (time 1767315322.54 total 90.11 last 35.44) total: 3402 link in 100.02 seconds: 34.01 ops/second using seed 2513203204 running for 100 seconds - stat 6349 (time 1767315244 ; total 11 ; last 11) - stat 10000 (time 1767315249 ; total 16 ; last 5) - stat 17613 (time 1767315260 ; total 27 ; last 11) - stat 20000 (time 1767315262 ; total 29 ; last 2) - stat 24224 (time 1767315273 ; total 40 ; last 11) - stat 29072 (time 1767315284 ; total 51 ; last 11) - stat 30000 (time 1767315287 ; total 54 ; last 3) - stat 33127 (time 1767315298 ; total 65 ; last 11) - stat 36369 (time 1767315309 ; total 76 ; last 11) - stat 39752 (time 1767315320 ; total 87 ; last 11) - stat 40000 (time 1767315320 ; total 87 ; last 0) - stat 43958 (time 1767315331 ; total 98 ; last 11) total: 44767 stats in 100 seconds: 447.670013 stats/second - unlinked 0 (time 1767315283 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3402) error: No such file or directory total: 3401 unlinks in 81 seconds: 41.987656 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (23344) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (23346) - No such process PASS 12 (139s) == sanityn test 13: test directory page revocation ======= 19:56:09 (1767315369) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:56:17 (1767315377) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 24529 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:56:24 (1767315384) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 25108 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:56:30 (1767315390) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25686 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:56:37 (1767315397) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 26268 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:56:43 (1767315403) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26850 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:56:53 (1767315413) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (47s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:57:39 (1767315459) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.469508 s, 17.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.298485 s, 28.1 MB/s Chance of close/open is 1 in 50 Seed set to 5463 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1767315463.858149 write 0x34533a thru 0x34df95 (0x8c5c bytes) 000008[0] 1767315463.937990 write 0x26cb40 thru 0x2769af (0x9e70 bytes) truncating to largest ever: 0x858741 000011[0] 1767315464.037582 trunc from 0x34df96 to 0x858740 (0x50a7ab bytes) 000012[1] 1767315464.064421 write 0x3c6013 thru 0x3cb7e0 (0x57ce bytes) 000015[0] 1767315464.136789 mapread 0x210c72 thru 0x212179 (0x1508 bytes) 000016[1] 1767315464.193868 mapwrite 0x1dcd55 thru 0x1dfe8d (0x3139 bytes) 000017[0] 1767315464.274639 read 0x839cce thru 0x8457b5 (0xbae8 bytes) 000018[0] 1767315464.323817 read 0x20e91d thru 0x20edcc (0x4b0 bytes) 000019[1] 1767315464.334999 write 0x6ec01a thru 0x6f3cb7 (0x7c9e bytes) 000020[0] 1767315464.356516 trunc from 0x858741 to 0x3c3fcb (0x494775 bytes) 000021[1] 1767315464.413149 mapwrite 0x1d8d71 thru 0x1df833 (0x6ac3 bytes) 000022[1] 1767315464.494802 mapwrite 0x8ac55b thru 0x8baacd (0xe573 bytes) 000023[1] 1767315464.629067 trunc from 0x8baace to 0x3b7230 (0x50389d bytes) 000024[0] 1767315464.702622 write 0x1946de thru 0x1960da (0x19fd bytes) 000025[0] 1767315464.756129 mapwrite 0x5a68b3 thru 0x5ad7b8 (0x6f06 bytes) 000027[0] 1767315464.918491 mapwrite 0x90fdfa thru 0x91de36 (0xe03d bytes) 000028[0] 1767315464.980335 read 0x6ab4c3 thru 0x6b0f03 (0x5a41 bytes) 000030[0] 1767315465.019368 mapwrite 0x18b357 thru 0x190c2a (0x58d4 bytes) 000031[1] 1767315465.064065 read 0x1849b9 thru 0x190891 (0xbed9 bytes) 000036[0] 1767315465.133888 mapwrite 0x5e5210 thru 0x5ee785 (0x9576 bytes) 000038[0] 1767315465.256295 mapwrite 0x0fe443 thru 0x0fe9d3 (0x591 bytes) 000039[0] 1767315465.385052 mapread 0x42b279 thru 0x43492f (0x96b7 bytes) 000040[1] 1767315465.438596 read 0x2b0f55 thru 0x2b3511 (0x25bd bytes) 000041[1] 1767315465.471616 read 0x2c8445 thru 0x2cca97 (0x4653 bytes) 000042[0] 1767315465.494717 mapwrite 0x99d0c0 thru 0x9a8c3f (0xbb80 bytes) 000043[1] 1767315465.631110 write 0x2077d9 thru 0x20b3d0 (0x3bf8 bytes) 000044[0] 1767315465.669670 mapwrite 0x149a21 thru 0x14ad93 (0x1373 bytes) 000045[1] 1767315465.744589 read 0x03ab7d thru 0x04150c (0x6990 bytes) 000047[0] 1767315465.784109 mapwrite 0x3544ae thru 0x35e655 (0xa1a8 bytes) 000048[0] 1767315465.861050 trunc from 0x9a8c40 to 0x7369e2 (0x27225d bytes) 000049[1] 1767315465.915450 mapread 0x0da0bc thru 0x0df396 (0x52db bytes) 000051[1] 1767315465.979907 read 0x4422ad thru 0x44fb7b (0xd8cf bytes) 000056[0] 1767315466.015162 mapread 0x263bfd thru 0x26bc91 (0x8095 bytes) 000058[0] 1767315466.039289 mapread 0x54e73a thru 0x55b725 (0xcfec bytes) 000059[1] 1767315466.102109 trunc from 0x7369e3 to 0x65b8b5 (0xdb12d bytes) 000060[1] 1767315466.186619 mapread 0x386a64 thru 0x388dd8 (0x2375 bytes) 000067[0] 1767315466.414214 write 0x4e99e7 thru 0x4f87b3 (0xedcd bytes) 000068[1] 1767315466.475960 read 0x62624a thru 0x635624 (0xf3db bytes) 000072[1] 1767315466.555624 mapread 0x145750 thru 0x14dae0 (0x8391 bytes) 000076[1] 1767315466.638623 mapwrite 0x728b14 thru 0x735516 (0xca03 bytes) 000078[0] 1767315466.921514 mapread 0x6b8741 thru 0x6b9359 (0xc19 bytes) 000079[0] 1767315467.015560 mapread 0x3fca1d thru 0x3fd95c (0xf40 bytes) 000080[1] 1767315467.053574 mapread 0x02aadc thru 0x031c87 (0x71ac bytes) 000082[0] 1767315467.102469 read 0x36d2fa thru 0x37c1e3 (0xeeea bytes) 000083[0] 1767315467.113302 trunc from 0x735517 to 0x71555e (0x1ffb8 bytes) 000084[1] 1767315467.254934 trunc from 0x71555f to 0x51dbfb (0x1f7963 bytes) 000086[0] 1767315467.371838 trunc from 0x51dbfc to 0x62d921 (0x10fd26 bytes) 000090[1] 1767315467.509728 mapwrite 0x9951f9 thru 0x9a3812 (0xe61a bytes) 000092[1] 1767315467.657169 write 0x5f7583 thru 0x604859 (0xd2d7 bytes) 000095[1] 1767315467.743296 mapread 0x27bfe2 thru 0x2835ec (0x760b bytes) 000096[1] 1767315467.768043 write 0x30650b thru 0x309001 (0x2af7 bytes) 000099[1] 1767315467.806178 mapread 0x242c78 thru 0x248bd3 (0x5f5c bytes) 000101[1] 1767315467.883700 mapread 0x3b5bff thru 0x3bcfb9 (0x73bb bytes) 000103[1] 1767315467.977125 mapread 0x7e05a5 thru 0x7e312d (0x2b89 bytes) 000104[1] 1767315467.999320 trunc from 0x9a3813 to 0x551f14 (0x4518fe bytes) 000107[1] 1767315468.030758 mapwrite 0x3c0412 thru 0x3c902c (0x8c1b bytes) 000108[0] 1767315468.117538 mapread 0x27361b thru 0x27db9d (0xa583 bytes) 000109[0] 1767315468.164711 mapread 0x3da6a3 thru 0x3ea3b4 (0xfd12 bytes) 000111[0] 1767315468.206224 read 0x2de334 thru 0x2e3555 (0x5222 bytes) 000113[0] 1767315468.221009 mapwrite 0x3b2346 thru 0x3b7372 (0x502d bytes) 000116[0] 1767315468.328527 read 0x3d305d thru 0x3dc73f (0x96e3 bytes) truncating to largest ever: 0x8f180a 000120[1] 1767315468.395263 trunc from 0x551f15 to 0x8f1809 (0x39f8f5 bytes) 000122[0] 1767315468.462495 write 0x0af886 thru 0x0bbe4b (0xc5c6 bytes) 000126[1] 1767315468.554002 mapwrite 0x769d79 thru 0x779316 (0xf59e bytes) 000128[1] 1767315468.693472 mapread 0x1ec554 thru 0x1fbb7f (0xf62c bytes) 000129[0] 1767315468.753548 trunc from 0x8f180a to 0x343555 (0x5ae2b4 bytes) 000132[0] 1767315468.819744 mapread 0x15f0e7 thru 0x16529d (0x61b7 bytes) 000141[1] 1767315468.957484 write 0x32ad75 thru 0x32e98f (0x3c1b bytes) 000145[0] 1767315469.041381 write 0x4048c0 thru 0x40ddd7 (0x9518 bytes) 000146[0] 1767315469.061257 write 0x68acb1 thru 0x694536 (0x9886 bytes) 000148[1] 1767315469.095318 trunc from 0x694537 to 0x8a9b58 (0x215622 bytes) 000150[0] 1767315469.187719 trunc from 0x8a9b59 to 0x2ca034 (0x5dfb24 bytes) 000153[1] 1767315469.274408 read 0x07d760 thru 0x084c85 (0x7526 bytes) 000154[1] 1767315469.293005 mapread 0x2c5314 thru 0x2c537e (0x6b bytes) 000155[0] 1767315469.302951 mapread 0x10fb24 thru 0x1186c8 (0x8ba5 bytes) 000156[1] 1767315469.331579 mapread 0x06f9f6 thru 0x07908e (0x9699 bytes) 000159[0] 1767315469.356971 trunc from 0x2ca035 to 0x3bc606 (0xf25d2 bytes) 000160[1] 1767315469.412928 mapwrite 0x9b71d6 thru 0x9c631e (0xf149 bytes) 000166[1] 1767315469.618534 trunc from 0x9c631f to 0x7134fc (0x2b2e22 bytes) 000167[0] 1767315469.647670 trunc from 0x7134fd to 0x065a1b (0x6adae1 bytes) 000170[0] 1767315469.735964 trunc from 0x065a1c to 0x6d816b (0x672750 bytes) 000173[1] 1767315469.786027 mapwrite 0x327668 thru 0x32a92e (0x32c7 bytes) 000174[0] 1767315469.885211 trunc from 0x6d816c to 0x0e60e8 (0x5f2083 bytes) 000177[0] 1767315469.980685 trunc from 0x0e60e9 to 0x3781b9 (0x2920d1 bytes) 000180[1] 1767315470.034503 read 0x23beb5 thru 0x24710f (0xb25b bytes) 000181[0] 1767315470.146826 read 0x028e4b thru 0x02eb48 (0x5cfe bytes) 000182[0] 1767315470.253450 read 0x0966e5 thru 0x098bae (0x24ca bytes) 000185[0] 1767315470.281998 trunc from 0x3781ba to 0x1b2eb4 (0x1c5305 bytes) 000187[1] 1767315470.352436 mapwrite 0x490e0b thru 0x498a53 (0x7c49 bytes) 000189[0] 1767315470.488676 trunc from 0x498a54 to 0x7cb3f4 (0x3329a1 bytes) 000191[1] 1767315470.548717 read 0x67bd6c thru 0x68427d (0x8512 bytes) 000192[0] 1767315470.604669 read 0x4f5f20 thru 0x502d00 (0xcde1 bytes) 000193[1] 1767315470.627755 mapread 0x50eb8f thru 0x51ea0c (0xfe7e bytes) 000194[1] 1767315470.685027 trunc from 0x7cb3f5 to 0x2b1264 (0x51a190 bytes) 000197[1] 1767315470.825229 write 0x9aa4d4 thru 0x9b1406 (0x6f33 bytes) 000200[1] 1767315470.856303 mapwrite 0x8580f1 thru 0x8588b2 (0x7c2 bytes) 000201[1] 1767315470.929092 mapwrite 0x67d6c1 thru 0x6846ab (0x6feb bytes) 000204[0] 1767315471.103522 trunc from 0x9b1407 to 0x67160b (0x33fdfb bytes) truncating to largest ever: 0x8fc835 000206[0] 1767315471.224006 trunc from 0x67160c to 0x8fc834 (0x28b229 bytes) 000211[1] 1767315471.301625 mapwrite 0x3d1840 thru 0x3dcc55 (0xb416 bytes) 000214[0] 1767315471.444106 mapread 0x4cbcdc thru 0x4d7180 (0xb4a5 bytes) 000215[0] 1767315471.503660 read 0x7d8265 thru 0x7dcba1 (0x493d bytes) 000216[1] 1767315471.536966 write 0x9ec5de thru 0x9f54e4 (0x8f07 bytes) 000217[0] 1767315471.552758 write 0x5a85c9 thru 0x5a9c8e (0x16c6 bytes) 000219[1] 1767315471.604117 mapwrite 0x5c91fb thru 0x5d40a0 (0xaea6 bytes) 000220[0] 1767315471.741808 trunc from 0x9f54e5 to 0x583040 (0x4724a4 bytes) 000226[0] 1767315471.937595 write 0x04b2fb thru 0x04c3f7 (0x10fd bytes) 000228[1] 1767315471.972455 write 0x211d1c thru 0x21abb0 (0x8e95 bytes) 000229[0] 1767315472.027764 read 0x28ac7e thru 0x2940da (0x945d bytes) 000230[1] 1767315472.112247 mapread 0x27b9b5 thru 0x283122 (0x776e bytes) 000231[1] 1767315472.168403 write 0x46d801 thru 0x46dc14 (0x414 bytes) 000232[0] 1767315472.241050 write 0x730569 thru 0x738a5d (0x84f5 bytes) 000233[0] 1767315472.311503 trunc from 0x738a5e to 0x54bbb4 (0x1ecea9 bytes) 000234[0] 1767315472.439817 read 0x4bdc7e thru 0x4bdddd (0x160 bytes) 000235[1] 1767315472.486946 mapread 0x4f11ad thru 0x4f7e60 (0x6cb4 bytes) 000236[0] 1767315472.578823 trunc from 0x54bbb5 to 0x4c191c (0x8a298 bytes) 000238[1] 1767315472.686353 mapwrite 0x2595a7 thru 0x268afa (0xf554 bytes) 000239[0] 1767315472.900861 write 0x342487 thru 0x34ec5f (0xc7d9 bytes) 000242[0] 1767315472.991930 read 0x395b1e thru 0x3a25a9 (0xca8c bytes) 000246[1] 1767315473.055952 trunc from 0x4c191d to 0x29a3d6 (0x227546 bytes) 000247[0] 1767315473.099988 mapwrite 0x647dac thru 0x64bc5b (0x3eb0 bytes) 000248[0] 1767315473.224234 read 0x44e191 thru 0x459f01 (0xbd71 bytes) 000250[1] 1767315473.278110 write 0x6136ab thru 0x615b21 (0x2477 bytes) 000251[1] 1767315473.310641 trunc from 0x64bc5c to 0x6c2d07 (0x770ac bytes) 000254[1] 1767315473.409087 write 0x1f8eee thru 0x202ba7 (0x9cba bytes) 000255[0] 1767315473.466054 mapwrite 0x8ef47a thru 0x8fd5d8 (0xe15f bytes) 000256[0] 1767315473.792516 read 0x780847 thru 0x783427 (0x2be1 bytes) 000257[1] 1767315473.821843 mapread 0x1b8314 thru 0x1c6f4d (0xec3a bytes) 000258[1] 1767315473.910373 mapwrite 0x5f2b08 thru 0x5f2b1f (0x18 bytes) 000260[1] 1767315474.067173 trunc from 0x8fd5d9 to 0x74431c (0x1b92bc bytes) 000261[1] 1767315474.127975 mapwrite 0x746361 thru 0x749e7d (0x3b1d bytes) 000265[1] 1767315474.320513 mapwrite 0x4ae0fe thru 0x4b8602 (0xa505 bytes) 000267[0] 1767315474.426265 read 0x70792a thru 0x708f96 (0x166d bytes) 000268[1] 1767315474.457704 read 0x11ba82 thru 0x11bdf1 (0x370 bytes) 000269[0] 1767315474.469938 mapwrite 0x3d875b thru 0x3d94bd (0xd63 bytes) 000270[0] 1767315474.550502 read 0x1655b1 thru 0x16dd74 (0x87c4 bytes) 000271[0] 1767315474.596615 trunc from 0x749e7e to 0x290d41 (0x4b913c bytes) 000273[1] 1767315474.746230 mapread 0x283047 thru 0x284f3e (0x1ef8 bytes) 000275[1] 1767315474.819250 mapread 0x12773f thru 0x12983f (0x2101 bytes) 000276[1] 1767315474.864851 write 0x34d6ae thru 0x35c635 (0xef88 bytes) 000279[1] 1767315474.943321 read 0x1a9598 thru 0x1b5f19 (0xc982 bytes) 000280[1] 1767315475.079985 mapread 0x0b28a6 thru 0x0b5448 (0x2ba3 bytes) 000282[1] 1767315475.130290 mapread 0x0be6d1 thru 0x0c2bed (0x451d bytes) 000288[1] 1767315475.318956 write 0x21cb9b thru 0x2210e6 (0x454c bytes) 000289[1] 1767315475.355997 mapread 0x09bb14 thru 0x09d4ec (0x19d9 bytes) 000290[0] 1767315475.426109 write 0x2c632e thru 0x2c9a09 (0x36dc bytes) 000292[0] 1767315475.499635 write 0x8df766 thru 0x8e8afc (0x9397 bytes) 000293[1] 1767315475.508161 read 0x40aa72 thru 0x4124e5 (0x7a74 bytes) 000298[1] 1767315475.620114 read 0x744d95 thru 0x74530e (0x57a bytes) 000300[1] 1767315475.657115 write 0x7760cf thru 0x776484 (0x3b6 bytes) 000301[1] 1767315475.683232 trunc from 0x8e8afd to 0x3ca815 (0x51e2e7 bytes) 000303[0] 1767315475.748935 write 0x6d46bf thru 0x6d724a (0x2b8c bytes) 000304[1] 1767315475.787120 mapread 0x592ee4 thru 0x59c5e4 (0x9701 bytes) 000305[1] 1767315475.851173 write 0x9be40a thru 0x9c52ba (0x6eb1 bytes) 000309[0] 1767315475.887002 write 0x083d17 thru 0x0928ae (0xeb98 bytes) 000310[1] 1767315475.915911 mapwrite 0x4eb37a thru 0x4f85da (0xd261 bytes) 000312[1] 1767315476.088704 mapwrite 0x94bee6 thru 0x95607b (0xa196 bytes) 000315[0] 1767315476.288081 write 0x862115 thru 0x86dcdd (0xbbc9 bytes) 000316[1] 1767315476.344879 read 0x4e63b7 thru 0x4e7482 (0x10cc bytes) 000317[1] 1767315476.359106 mapwrite 0x7bc359 thru 0x7c6e98 (0xab40 bytes) truncating to largest ever: 0x9ad42f 000318[1] 1767315476.635534 trunc from 0x9c52bb to 0x9ad42e (0x17e8c bytes) 000322[0] 1767315476.698127 trunc from 0x9ad42f to 0x420b75 (0x58c8b9 bytes) 000323[0] 1767315476.742521 mapread 0x12ce28 thru 0x132ffd (0x61d6 bytes) 000324[1] 1767315476.796540 mapread 0x1cdce3 thru 0x1d0bdc (0x2efa bytes) 000326[1] 1767315476.876796 mapread 0x38e6e0 thru 0x39b7c9 (0xd0ea bytes) 000327[0] 1767315476.917410 write 0x913763 thru 0x91a643 (0x6ee1 bytes) 000330[1] 1767315476.998613 mapread 0x36e982 thru 0x374d97 (0x6416 bytes) 000331[1] 1767315477.060264 trunc from 0x91a644 to 0x487999 (0x492caa bytes) 000332[0] 1767315477.110244 mapread 0x183ad3 thru 0x187bc0 (0x40ee bytes) 000333[0] 1767315477.140063 mapwrite 0x5f28e6 thru 0x5fe080 (0xb79b bytes) 000335[0] 1767315477.343877 write 0x777590 thru 0x778549 (0xfba bytes) 000336[1] 1767315477.356978 read 0x57fb22 thru 0x58eb1e (0xeffd bytes) 000338[1] 1767315477.419841 trunc from 0x77854a to 0x96ac0e (0x1f26c5 bytes) 000341[0] 1767315477.561399 mapread 0x8bbba8 thru 0x8c5551 (0x99aa bytes) 000342[1] 1767315477.643870 mapwrite 0x4db1d2 thru 0x4e9a0f (0xe83e bytes) 000346[0] 1767315477.825565 mapwrite 0x02059b thru 0x02ed9f (0xe805 bytes) 000347[1] 1767315477.992688 mapread 0x95e766 thru 0x95fc21 (0x14bc bytes) 000349[0] 1767315478.089775 trunc from 0x96ac0f to 0x823551 (0x1476bd bytes) 000351[0] 1767315478.155030 mapread 0x22f861 thru 0x23dfda (0xe77a bytes) 000354[1] 1767315478.206112 mapread 0x364ea2 thru 0x36644d (0x15ac bytes) 000357[0] 1767315478.275436 write 0x0c013d thru 0x0c57fe (0x56c2 bytes) 000361[1] 1767315478.347447 write 0x86149f thru 0x866427 (0x4f89 bytes) 000362[1] 1767315478.378871 read 0x23c6db thru 0x24727c (0xaba2 bytes) 000363[0] 1767315478.399408 trunc from 0x866428 to 0x9abb97 (0x145770 bytes) 000364[0] 1767315478.458985 mapread 0x0f9701 thru 0x10285e (0x915e bytes) 000366[0] 1767315478.499975 read 0x767138 thru 0x775519 (0xe3e2 bytes) 000368[0] 1767315478.526993 mapwrite 0x905755 thru 0x90b87a (0x6126 bytes) 000370[1] 1767315478.622048 trunc from 0x9abb98 to 0x1afcad (0x7fbeea bytes) truncating to largest ever: 0x9be27a 000371[1] 1767315478.667093 trunc from 0x1afcae to 0x9be279 (0x80e5cc bytes) 000372[1] 1767315478.707136 mapwrite 0x67a8ad thru 0x685bc3 (0xb317 bytes) 000373[0] 1767315478.851600 trunc from 0x9be27a to 0x554d07 (0x469572 bytes) 000374[0] 1767315478.926459 mapread 0x333c40 thru 0x3397da (0x5b9b bytes) 000377[1] 1767315479.018481 read 0x1b4d1b thru 0x1be13a (0x9420 bytes) 000378[1] 1767315479.068254 write 0x0345d4 thru 0x03cd72 (0x879f bytes) 000380[0] 1767315479.113040 trunc from 0x554d08 to 0x730e5d (0x1dc156 bytes) 000382[0] 1767315479.191109 trunc from 0x730e5e to 0x1d2bba (0x55e2a3 bytes) 000383[0] 1767315479.213304 trunc from 0x1d2bbb to 0x528138 (0x35557e bytes) 000384[1] 1767315479.229885 read 0x37589e thru 0x37f373 (0x9ad6 bytes) 000386[1] 1767315479.301987 trunc from 0x528139 to 0x9bc835 (0x4946fd bytes) 000389[0] 1767315479.364752 trunc from 0x9bc836 to 0x2bff6f (0x6fc8c6 bytes) 000390[1] 1767315479.434482 write 0x043228 thru 0x04f7a0 (0xc579 bytes) 000397[1] 1767315479.633178 read 0x00393c thru 0x0089ba (0x507f bytes) 000398[1] 1767315479.721449 read 0x092051 thru 0x094647 (0x25f7 bytes) 000399[0] 1767315479.744807 mapread 0x00ac16 thru 0x019735 (0xeb20 bytes) 000400[0] 1767315479.849812 trunc from 0x2bff70 to 0x3f2e50 (0x132ee1 bytes) 000403[1] 1767315479.990466 write 0x8c3eb6 thru 0x8c3f64 (0xaf bytes) 000409[0] 1767315480.138951 write 0x9bd0ea thru 0x9cc820 (0xf737 bytes) 000410[0] 1767315480.155236 read 0x92072a thru 0x92c9a1 (0xc278 bytes) 000412[1] 1767315480.196498 mapwrite 0x6eadf4 thru 0x6f0831 (0x5a3e bytes) 000415[0] 1767315480.517760 mapread 0x6b9ab9 thru 0x6bf5f6 (0x5b3e bytes) 000418[1] 1767315480.583117 mapwrite 0x97986a thru 0x97b228 (0x19bf bytes) 000420[1] 1767315480.869391 mapwrite 0x3b4205 thru 0x3c28ac (0xe6a8 bytes) 000422[1] 1767315480.940725 mapread 0x731d0d thru 0x735fef (0x42e3 bytes) 000424[0] 1767315481.000903 mapwrite 0x844cf7 thru 0x852ac8 (0xddd2 bytes) 000431[0] 1767315481.349111 mapwrite 0x80bf6c thru 0x80dd2d (0x1dc2 bytes) 000433[0] 1767315481.445882 trunc from 0x9cc821 to 0x90c06b (0xc07b5 bytes) 000434[1] 1767315481.517788 mapwrite 0x6de4cc thru 0x6ee374 (0xfea9 bytes) 000435[1] 1767315481.814564 read 0x67742c thru 0x6809b4 (0x9589 bytes) 000437[1] 1767315481.910661 mapread 0x051d5c thru 0x056790 (0x4a35 bytes) truncating to largest ever: 0x9e05d8 000439[1] 1767315481.991962 trunc from 0x90c06c to 0x9e05d7 (0xd456c bytes) 000440[1] 1767315482.101981 write 0x9ccd90 thru 0x9d1785 (0x49f6 bytes) 000441[1] 1767315482.171107 mapread 0x49728f thru 0x49cddb (0x5b4d bytes) 000442[1] 1767315482.247505 mapread 0x844e69 thru 0x848a0d (0x3ba5 bytes) 000443[0] 1767315482.371278 mapread 0x770bbb thru 0x77e8b2 (0xdcf8 bytes) 000444[0] 1767315482.465385 trunc from 0x9e05d8 to 0x0d5762 (0x90ae75 bytes) 000445[0] 1767315482.556576 mapread 0x07bfbb thru 0x087a7a (0xbac0 bytes) 000449[1] 1767315482.710385 mapwrite 0x55af33 thru 0x56a42f (0xf4fd bytes) 000452[1] 1767315482.948668 mapwrite 0x6735b7 thru 0x673c71 (0x6bb bytes) 000453[0] 1767315483.072312 write 0x20f9c1 thru 0x214d6f (0x53af bytes) 000454[0] 1767315483.138153 trunc from 0x673c72 to 0x15117d (0x522af4 bytes) 000455[0] 1767315483.220838 mapread 0x0a8c73 thru 0x0b03e2 (0x7770 bytes) 000456[1] 1767315483.271285 trunc from 0x15117e to 0x29ea72 (0x14d8f5 bytes) 000458[0] 1767315483.374380 trunc from 0x29ea73 to 0x81396f (0x574efd bytes) 000459[0] 1767315483.449536 read 0x38c7d5 thru 0x396f51 (0xa77d bytes) 000462[0] 1767315483.507774 mapwrite 0x726b78 thru 0x72b31a (0x47a3 bytes) 000465[1] 1767315483.709299 mapread 0x4cc144 thru 0x4ce2e5 (0x21a2 bytes) 000468[1] 1767315483.818493 read 0x4c2d8b thru 0x4c779f (0x4a15 bytes) 000470[0] 1767315483.826781 trunc from 0x813970 to 0x90f7a4 (0xfbe35 bytes) 000478[1] 1767315483.969764 read 0x356fe9 thru 0x36017e (0x9196 bytes) 000481[0] 1767315484.008178 read 0x77a026 thru 0x785cb4 (0xbc8f bytes) 000482[1] 1767315484.041261 trunc from 0x90f7a5 to 0x0cd253 (0x842551 bytes) 000483[0] 1767315484.115759 read 0x058bc6 thru 0x05aa02 (0x1e3d bytes) 000484[1] 1767315484.173882 write 0x1fefcb thru 0x205b7f (0x6bb5 bytes) 000486[1] 1767315484.198234 mapread 0x1cc8e5 thru 0x1d9ac3 (0xd1df bytes) 000487[0] 1767315484.250290 read 0x1092f0 thru 0x10d903 (0x4614 bytes) 000490[0] 1767315484.374665 trunc from 0x205b80 to 0x0efa63 (0x11611c bytes) 000491[0] 1767315484.457754 write 0x514daf thru 0x519484 (0x46d6 bytes) 000492[0] 1767315484.521484 mapwrite 0x656ecc thru 0x660510 (0x9645 bytes) 000494[1] 1767315484.912830 write 0x987231 thru 0x996319 (0xf0e9 bytes) 000496[1] 1767315485.034412 mapread 0x3668d6 thru 0x373a5f (0xd18a bytes) 000497[1] 1767315485.120173 mapread 0x89bb95 thru 0x8a87ed (0xcc59 bytes) 000499[0] 1767315485.211420 read 0x316521 thru 0x31701f (0xaff bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 5485 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x5f23d 000011[1] 1767315486.414971 trunc from 00000000 to 0x05f23c (0x5f23d bytes) 000013[0] 1767315486.447539 read 0x00c000 thru 0x013ad7 (0x7ad8 bytes) 000015[1] 1767315486.470822 mapread 0x04c000 thru 0x050bff (0x4c00 bytes) 000016[1] 1767315486.486511 read 0x003000 thru 0x00ae57 (0x7e58 bytes) 000018[0] 1767315486.494358 read 0x049000 thru 0x055cbc (0xccbd bytes) 000019[0] 1767315486.519577 write 0x6e9000 thru 0x6ec79f (0x37a0 bytes) truncating to largest ever: 0x5b9b84 000020[1] 1767315486.576047 trunc from 0x6ec7a0 to 0x5b9b83 (0x132c1c bytes) 000021[0] 1767315486.657382 read 0x237000 thru 0x245bc5 (0xebc6 bytes) 000027[1] 1767315486.691936 write 0x97a000 thru 0x9870df (0xd0e0 bytes) 000028[1] 1767315486.730659 write 0x159000 thru 0x1634ea (0xa4eb bytes) 000030[1] 1767315486.739719 trunc from 0x9870e0 to 0x124a37 (0x8626a8 bytes) 000033[1] 1767315486.766697 mapread 0x117000 thru 0x119d1f (0x2d20 bytes) 000040[0] 1767315486.813530 mapread 0x08c000 thru 0x08faed (0x3aee bytes) 000042[0] 1767315486.883896 trunc from 0x124a38 to 0x060d56 (0xc3ce1 bytes) 000043[0] 1767315486.950380 write 0x645000 thru 0x64cbef (0x7bf0 bytes) 000044[0] 1767315486.998037 read 0x1fb000 thru 0x201fe7 (0x6fe8 bytes) 000045[0] 1767315487.011596 write 0x58f000 thru 0x59a849 (0xb84a bytes) truncating to largest ever: 0x67355b 000046[1] 1767315487.077162 trunc from 0x64cbf0 to 0x67355a (0x2696b bytes) 000047[0] 1767315487.161486 mapread 0x4b0000 thru 0x4bc7ce (0xc7cf bytes) 000049[1] 1767315487.233868 read 0x263000 thru 0x26869a (0x569b bytes) 000052[0] 1767315487.244768 mapread 0x492000 thru 0x493efb (0x1efc bytes) 000055[0] 1767315487.260504 mapwrite 0x012000 thru 0x01fdb6 (0xddb7 bytes) 000057[1] 1767315487.462413 read 0x28c000 thru 0x299756 (0xd757 bytes) 000059[1] 1767315487.493951 write 0x3f9000 thru 0x4085c7 (0xf5c8 bytes) 000062[1] 1767315487.626688 trunc from 0x67355b to 0x097963 (0x5dbbf7 bytes) 000063[1] 1767315487.686537 read 0x012000 thru 0x013d0f (0x1d10 bytes) 000066[0] 1767315487.750416 write 0x2d1000 thru 0x2d7494 (0x6495 bytes) 000068[0] 1767315487.815957 write 0x606000 thru 0x612320 (0xc321 bytes) 000070[0] 1767315487.841097 mapread 0x07e000 thru 0x08a108 (0xc109 bytes) 000075[1] 1767315487.970571 read 0x137000 thru 0x142a44 (0xba45 bytes) 000079[1] 1767315488.028527 mapwrite 0x92a000 thru 0x92b8a4 (0x18a5 bytes) 000080[0] 1767315488.205162 read 0x147000 thru 0x14bd87 (0x4d88 bytes) 000082[1] 1767315488.243163 write 0x58a000 thru 0x58ab1c (0xb1d bytes) 000083[1] 1767315488.332071 mapwrite 0x747000 thru 0x750db0 (0x9db1 bytes) 000085[0] 1767315488.438018 mapwrite 0x7ef000 thru 0x7fce0c (0xde0d bytes) 000086[0] 1767315488.669883 mapwrite 0x938000 thru 0x947dde (0xfddf bytes) 000087[1] 1767315488.939498 mapwrite 0x9d6000 thru 0x9d8303 (0x2304 bytes) 000088[1] 1767315489.086757 write 0x88d000 thru 0x89cdc2 (0xfdc3 bytes) 000090[1] 1767315489.102014 read 0x197000 thru 0x197402 (0x403 bytes) 000095[0] 1767315489.148924 mapwrite 0x556000 thru 0x5651ab (0xf1ac bytes) 000097[0] 1767315489.332133 mapwrite 0x5aa000 thru 0x5b88fa (0xe8fb bytes) truncating to largest ever: 0x722c8d 000098[0] 1767315489.447224 trunc from 0x9d8304 to 0x722c8c (0x2b5677 bytes) 000099[1] 1767315489.495884 trunc from 0x722c8d to 0x6994f1 (0x8979b bytes) 000100[1] 1767315489.525415 mapwrite 0x4c0000 thru 0x4c5091 (0x5092 bytes) 000101[1] 1767315489.627799 trunc from 0x6994f2 to 0x3eb659 (0x2ade98 bytes) 000102[0] 1767315489.644473 read 0x1e9000 thru 0x1e9ce6 (0xce7 bytes) 000103[1] 1767315489.692408 mapwrite 0x680000 thru 0x68f63e (0xf63f bytes) 000104[1] 1767315489.922822 read 0x294000 thru 0x2943b7 (0x3b8 bytes) 000113[0] 1767315490.021455 write 0x9e8000 thru 0x9f62fb (0xe2fc bytes) 000114[1] 1767315490.036076 write 0x4b8000 thru 0x4c6ff8 (0xeff9 bytes) 000115[0] 1767315490.071052 trunc from 0x9f62fc to 0x6f894a (0x2fd9b1 bytes) 000116[1] 1767315490.142737 read 0x5b5000 thru 0x5c3747 (0xe748 bytes) 000117[0] 1767315490.169943 trunc from 0x6f894b to 0x48f81f (0x26912b bytes) truncating to largest ever: 0x81bc30 000118[0] 1767315490.213686 trunc from 0x48f820 to 0x81bc2f (0x38c410 bytes) 000122[0] 1767315490.247712 read 0x3e7000 thru 0x3f5a83 (0xea84 bytes) 000123[0] 1767315490.256747 mapread 0x27c000 thru 0x28807f (0xc080 bytes) 000126[1] 1767315490.315567 mapwrite 0x370000 thru 0x37ea27 (0xea28 bytes) 000130[0] 1767315490.553832 trunc from 0x81bc30 to 0x66e09b (0x1adb94 bytes) 000138[1] 1767315490.687145 write 0x70d000 thru 0x715877 (0x8878 bytes) 000139[0] 1767315490.712205 mapread 0x567000 thru 0x5751f0 (0xe1f1 bytes) 000140[0] 1767315490.763838 write 0x674000 thru 0x67baa9 (0x7aaa bytes) 000141[0] 1767315490.789073 read 0x2af000 thru 0x2bb547 (0xc548 bytes) 000142[0] 1767315490.799070 read 0x32e000 thru 0x3311ad (0x31ae bytes) 000143[1] 1767315490.815000 mapwrite 0x05b000 thru 0x05cf94 (0x1f95 bytes) 000144[1] 1767315490.909287 mapread 0x206000 thru 0x2069a6 (0x9a7 bytes) 000146[0] 1767315490.928116 read 0x453000 thru 0x45e5aa (0xb5ab bytes) 000148[1] 1767315490.953500 write 0x425000 thru 0x430fab (0xbfac bytes) 000156[1] 1767315490.998489 write 0x297000 thru 0x29da33 (0x6a34 bytes) 000158[0] 1767315491.003921 trunc from 0x715878 to 0x30f2c7 (0x4065b0 bytes) 000159[0] 1767315491.064668 write 0x3f4000 thru 0x3f4cfb (0xcfc bytes) 000160[0] 1767315491.079009 mapwrite 0x0d8000 thru 0x0e3228 (0xb229 bytes) 000163[1] 1767315491.163093 write 0x9db000 thru 0x9e45e2 (0x95e3 bytes) 000164[0] 1767315491.178974 read 0x215000 thru 0x2246fa (0xf6fb bytes) 000165[1] 1767315491.302981 trunc from 0x9e45e3 to 0x74d68f (0x296f53 bytes) 000167[0] 1767315491.371154 read 0x6d3000 thru 0x6d68b2 (0x38b3 bytes) 000168[0] 1767315491.392567 read 0x108000 thru 0x1166ff (0xe700 bytes) 000170[1] 1767315491.409769 trunc from 0x74d690 to 0x0da08b (0x673604 bytes) 000171[1] 1767315491.465417 write 0x19b000 thru 0x1a5738 (0xa739 bytes) 000174[1] 1767315491.488832 mapwrite 0x389000 thru 0x3929ff (0x9a00 bytes) 000175[0] 1767315491.666444 trunc from 0x392a00 to 0x1abfef (0x1e6a10 bytes) 000177[1] 1767315491.714064 mapread 0x181000 thru 0x18d9e7 (0xc9e8 bytes) 000178[0] 1767315491.780114 mapwrite 0x1ee000 thru 0x1efbe2 (0x1be3 bytes) truncating to largest ever: 0x850bbd 000179[0] 1767315491.853077 trunc from 0x1efbe3 to 0x850bbc (0x660fda bytes) 000183[1] 1767315491.910044 mapread 0x7a2000 thru 0x7a7de5 (0x5de6 bytes) 000184[0] 1767315491.952515 mapread 0x1db000 thru 0x1db6ba (0x6bb bytes) 000186[1] 1767315491.966124 mapread 0x5d8000 thru 0x5e7e30 (0xfe31 bytes) 000190[0] 1767315491.996527 trunc from 0x850bbd to 0x75b017 (0xf5ba5 bytes) 000191[1] 1767315492.054953 read 0x16a000 thru 0x175b19 (0xbb1a bytes) 000193[0] 1767315492.090471 mapread 0x70f000 thru 0x71ef39 (0xff3a bytes) 000194[1] 1767315492.116439 write 0x160000 thru 0x16d41e (0xd41f bytes) 000197[0] 1767315492.198023 read 0x30a000 thru 0x310915 (0x6916 bytes) 000198[0] 1767315492.243828 mapread 0x4fd000 thru 0x500e2d (0x3e2e bytes) 000199[1] 1767315492.271072 mapwrite 0x74c000 thru 0x759ee4 (0xdee5 bytes) 000200[1] 1767315492.432851 read 0x6ba000 thru 0x6bff6f (0x5f70 bytes) 000202[1] 1767315492.458963 mapread 0x6ac000 thru 0x6b2d07 (0x6d08 bytes) 000203[1] 1767315492.470735 trunc from 0x75b018 to 0x0339e3 (0x727634 bytes) 000208[0] 1767315492.531552 mapwrite 0x772000 thru 0x772b57 (0xb58 bytes) 000209[0] 1767315492.639619 trunc from 0x772b58 to 0x768c86 (0x9ed1 bytes) 000210[1] 1767315492.680717 mapread 0x1f5000 thru 0x1f7157 (0x2158 bytes) 000211[1] 1767315492.726763 read 0x1d1000 thru 0x1da875 (0x9876 bytes) 000212[1] 1767315492.727453 trunc from 0x768c87 to 0x6ab122 (0xbdb64 bytes) 000218[1] 1767315492.823567 trunc from 0x6ab123 to 0x6b1470 (0x634e bytes) 000220[1] 1767315492.862685 trunc from 0x6b1471 to 0x2cf87a (0x3e1bf6 bytes) 000224[1] 1767315492.929582 mapwrite 0x620000 thru 0x62b433 (0xb434 bytes) 000225[0] 1767315493.067440 mapwrite 0x025000 thru 0x032151 (0xd152 bytes) 000226[1] 1767315493.197686 mapwrite 0x6fa000 thru 0x6fd8d9 (0x38da bytes) 000229[0] 1767315493.324531 write 0x31d000 thru 0x322092 (0x5093 bytes) 000231[1] 1767315493.342345 write 0x7dd000 thru 0x7e0cbb (0x3cbc bytes) 000233[0] 1767315493.372488 mapread 0x71f000 thru 0x71f5d3 (0x5d4 bytes) 000234[1] 1767315493.402453 read 0x113000 thru 0x11aad7 (0x7ad8 bytes) 000238[0] 1767315493.410220 write 0x26e000 thru 0x2797d4 (0xb7d5 bytes) 000243[0] 1767315493.454361 read 0x763000 thru 0x76a6f1 (0x76f2 bytes) 000247[1] 1767315493.515526 mapread 0x3ff000 thru 0x408707 (0x9708 bytes) 000248[0] 1767315493.580061 mapread 0x494000 thru 0x495b53 (0x1b54 bytes) 000250[0] 1767315493.598838 trunc from 0x7e0cbc to 0x29015c (0x550b5f bytes) 000251[1] 1767315493.644306 trunc from 0x29015d to 0x2381a4 (0x57fb8 bytes) 000252[0] 1767315493.669828 mapread 0x1bf000 thru 0x1bf196 (0x197 bytes) 000261[1] 1767315493.697993 mapwrite 0x89f000 thru 0x8a24a3 (0x34a4 bytes) 000264[1] 1767315493.809497 mapread 0x3a7000 thru 0x3b1310 (0xa311 bytes) 000265[0] 1767315493.841695 write 0x952000 thru 0x952c50 (0xc51 bytes) 000266[1] 1767315493.874673 read 0x256000 thru 0x25685e (0x85f bytes) 000267[1] 1767315493.904448 read 0x609000 thru 0x616c8c (0xdc8d bytes) 000271[0] 1767315493.919713 trunc from 0x952c51 to 0x0b760b (0x89b645 bytes) 000272[1] 1767315493.991950 write 0x067000 thru 0x06e461 (0x7462 bytes) 000273[1] 1767315494.041121 write 0x2dd000 thru 0x2dd997 (0x998 bytes) 000274[0] 1767315494.059842 mapwrite 0x986000 thru 0x98a097 (0x4098 bytes) 000277[0] 1767315494.200694 mapwrite 0x290000 thru 0x2900a5 (0xa6 bytes) 000281[0] 1767315494.308727 write 0x416000 thru 0x419e04 (0x3e05 bytes) 000284[1] 1767315494.353725 mapread 0x94f000 thru 0x956721 (0x7722 bytes) 000286[0] 1767315494.397797 trunc from 0x98a098 to 0x1c69de (0x7c36b9 bytes) 000287[0] 1767315494.442025 read 0x04c000 thru 0x05471f (0x8720 bytes) 000288[0] 1767315494.520649 read 0x022000 thru 0x02e605 (0xc606 bytes) 000289[0] 1767315494.526667 trunc from 0x1c69df to 0x453be3 (0x28d205 bytes) 000292[1] 1767315494.569972 write 0x424000 thru 0x4264b8 (0x24b9 bytes) 000294[0] 1767315494.592655 write 0x7da000 thru 0x7e712d (0xd12e bytes) 000295[1] 1767315494.624681 mapwrite 0x9fa000 thru 0x9fb078 (0x1079 bytes) 000296[0] 1767315494.715037 mapwrite 0x02a000 thru 0x038f8b (0xef8c bytes) 000298[1] 1767315494.880377 mapread 0x3d4000 thru 0x3e39df (0xf9e0 bytes) 000299[0] 1767315494.935298 mapread 0x6af000 thru 0x6bce9f (0xdea0 bytes) 000300[1] 1767315495.020743 mapwrite 0x97a000 thru 0x989f2a (0xff2b bytes) 000301[0] 1767315495.104701 trunc from 0x9fb079 to 0x3cba70 (0x62f608 bytes) 000302[0] 1767315495.130983 trunc from 0x3cba71 to 0x7e657f (0x41ab0f bytes) 000303[0] 1767315495.143297 write 0x061000 thru 0x064aa1 (0x3aa2 bytes) 000305[0] 1767315495.177393 write 0x7c6000 thru 0x7d2cab (0xccac bytes) 000306[0] 1767315495.193910 read 0x053000 thru 0x05f832 (0xc833 bytes) 000308[1] 1767315495.226448 mapwrite 0x7a6000 thru 0x7ab75a (0x575b bytes) 000309[1] 1767315495.325474 mapwrite 0x392000 thru 0x397ff8 (0x5ff9 bytes) 000310[1] 1767315495.382962 mapread 0x109000 thru 0x10d205 (0x4206 bytes) 000311[1] 1767315495.397801 mapwrite 0x5d0000 thru 0x5d173c (0x173d bytes) 000312[1] 1767315495.474897 mapwrite 0x4b1000 thru 0x4b94bc (0x84bd bytes) 000314[1] 1767315495.569900 mapwrite 0x31d000 thru 0x31f422 (0x2423 bytes) 000318[0] 1767315495.703959 mapwrite 0x572000 thru 0x577f75 (0x5f76 bytes) 000320[1] 1767315495.930839 write 0x256000 thru 0x25b3b1 (0x53b2 bytes) 000321[1] 1767315496.007299 trunc from 0x7e6580 to 0x1d528a (0x6112f5 bytes) 000322[1] 1767315496.055467 trunc from 0x1d528b to 0x4aa79f (0x2d5515 bytes) 000323[1] 1767315496.109063 mapread 0x48a000 thru 0x48f875 (0x5876 bytes) 000326[1] 1767315496.194356 mapread 0x249000 thru 0x2522ea (0x92eb bytes) 000329[1] 1767315496.221632 mapwrite 0x871000 thru 0x873883 (0x2884 bytes) 000330[1] 1767315496.437961 mapwrite 0x50a000 thru 0x514de4 (0xade5 bytes) 000332[1] 1767315496.557056 write 0x1be000 thru 0x1c25e4 (0x45e5 bytes) 000335[1] 1767315496.564727 write 0x46a000 thru 0x46e21b (0x421c bytes) 000341[0] 1767315496.616021 trunc from 0x873884 to 0x057e38 (0x81ba4b bytes) 000342[1] 1767315496.705587 write 0x8c9000 thru 0x8cbb91 (0x2b92 bytes) 000343[0] 1767315496.745521 trunc from 0x8cbb92 to 0x2bf6ae (0x60c4e3 bytes) 000344[0] 1767315496.831839 write 0x037000 thru 0x03d5b5 (0x65b6 bytes) 000346[1] 1767315496.875652 write 0x1ad000 thru 0x1bae16 (0xde17 bytes) 000349[1] 1767315496.928464 mapread 0x296000 thru 0x29b2ab (0x52ac bytes) 000351[1] 1767315496.979928 trunc from 0x2bf6af to 0x45b31d (0x19bc6f bytes) 000352[0] 1767315497.016459 write 0x9fb000 thru 0x9ff592 (0x4593 bytes) 000354[0] 1767315497.070780 read 0x3c0000 thru 0x3c8f47 (0x8f48 bytes) 000355[1] 1767315497.100025 mapwrite 0x609000 thru 0x609458 (0x459 bytes) 000357[0] 1767315497.200352 read 0x01d000 thru 0x025221 (0x8222 bytes) 000361[1] 1767315497.221061 mapwrite 0x1b5000 thru 0x1c2248 (0xd249 bytes) 000362[1] 1767315497.320269 trunc from 0x9ff593 to 0x6ef8ae (0x30fce4 bytes) 000363[1] 1767315497.349749 read 0x322000 thru 0x327259 (0x525a bytes) 000366[0] 1767315497.404010 read 0x5d5000 thru 0x5dcc84 (0x7c85 bytes) 000368[0] 1767315497.434247 write 0x700000 thru 0x70ea0e (0xea0f bytes) 000370[1] 1767315497.505875 write 0x43b000 thru 0x43d0fb (0x20fc bytes) 000371[1] 1767315497.540719 write 0x240000 thru 0x242272 (0x2273 bytes) 000374[0] 1767315497.605500 write 0x0ee000 thru 0x0f011a (0x211b bytes) 000375[1] 1767315497.675635 mapwrite 0x2ef000 thru 0x2f7af9 (0x8afa bytes) 000377[1] 1767315497.858941 mapread 0x591000 thru 0x5999bf (0x89c0 bytes) 000379[0] 1767315497.988590 trunc from 0x70ea0f to 0x779287 (0x6a879 bytes) 000381[0] 1767315498.098429 write 0x04c000 thru 0x059306 (0xd307 bytes) 000382[1] 1767315498.110915 mapwrite 0x2dd000 thru 0x2e2066 (0x5067 bytes) 000383[0] 1767315498.332347 mapwrite 0x61f000 thru 0x6234e6 (0x44e7 bytes) 000385[1] 1767315498.525806 write 0x1a3000 thru 0x1ad194 (0xa195 bytes) 000388[0] 1767315498.599070 write 0x5ef000 thru 0x5f0e5a (0x1e5b bytes) 000389[0] 1767315498.709871 mapread 0x2c3000 thru 0x2cc258 (0x9259 bytes) 000391[1] 1767315498.779786 read 0x6d4000 thru 0x6d7f09 (0x3f0a bytes) 000393[0] 1767315498.818156 write 0x4e7000 thru 0x4e9efe (0x2eff bytes) 000394[1] 1767315498.887059 mapread 0x1d8000 thru 0x1dec65 (0x6c66 bytes) 000399[0] 1767315498.943522 mapread 0x58a000 thru 0x58e4ff (0x4500 bytes) 000400[0] 1767315498.959484 write 0x956000 thru 0x95f726 (0x9727 bytes) 000401[0] 1767315498.972062 mapread 0x074000 thru 0x083259 (0xf25a bytes) 000404[1] 1767315499.033864 read 0x5cc000 thru 0x5d02a9 (0x42aa bytes) 000406[1] 1767315499.067473 read 0x128000 thru 0x13401e (0xc01f bytes) 000407[0] 1767315499.083073 read 0x45a000 thru 0x45f025 (0x5026 bytes) 000408[0] 1767315499.095723 trunc from 0x95f727 to 0x84345c (0x11c2ca bytes) 000409[1] 1767315499.135702 write 0x0d7000 thru 0x0dce68 (0x5e69 bytes) 000410[1] 1767315499.151777 write 0x0fc000 thru 0x104331 (0x8332 bytes) 000411[1] 1767315499.177134 write 0x90d000 thru 0x916606 (0x9607 bytes) 000413[1] 1767315499.206275 mapwrite 0x35b000 thru 0x3660f6 (0xb0f7 bytes) 000415[1] 1767315499.419574 mapwrite 0x015000 thru 0x0162f1 (0x12f2 bytes) 000416[1] 1767315499.477228 write 0x354000 thru 0x36266b (0xe66c bytes) 000418[1] 1767315499.505055 mapwrite 0x310000 thru 0x312c3e (0x2c3f bytes) 000419[0] 1767315499.657495 mapwrite 0x001000 thru 0x004686 (0x3687 bytes) 000422[0] 1767315499.844556 trunc from 0x916607 to 0x6b35df (0x263027 bytes) 000425[0] 1767315499.929151 write 0x988000 thru 0x9896c4 (0x16c5 bytes) 000430[1] 1767315500.006231 trunc from 0x9896c5 to 0x5c6872 (0x3c2e52 bytes) 000436[0] 1767315500.278993 mapread 0x053000 thru 0x05cf5d (0x9f5e bytes) 000438[0] 1767315500.356878 trunc from 0x5c6873 to 0x801419 (0x23aba7 bytes) truncating to largest ever: 0x9ac1b3 000439[1] 1767315500.414714 trunc from 0x80141a to 0x9ac1b2 (0x1aad99 bytes) 000440[0] 1767315500.502881 write 0x90b000 thru 0x90e63c (0x363d bytes) 000441[0] 1767315500.538747 mapread 0x5ef000 thru 0x5f7282 (0x8283 bytes) 000444[0] 1767315500.599061 read 0x92c000 thru 0x92e4cc (0x24cd bytes) 000445[1] 1767315500.633306 write_OD 0x779000 thru 0x77ffff (0x7000 bytes) 000446[1] 1767315500.710697 write_OD 0x5c4000 thru 0x5cefff (0xb000 bytes) 000450[1] 1767315500.811907 read_OD 0x67a000 thru 0x683fff (0xa000 bytes) 000454[0] 1767315500.820962 write 0x737000 thru 0x7452e1 (0xe2e2 bytes) 000455[0] 1767315500.855179 mapwrite 0x799000 thru 0x7a3043 (0xa044 bytes) 000456[1] 1767315500.933456 read_OD 0x00b000 thru 0x012fff (0x8000 bytes) 000457[1] 1767315500.964229 read_OD 0x3f7000 thru 0x400fff (0xa000 bytes) 000460[1] 1767315500.978090 trunc from 0x9ac1b3 to 0x776d5d (0x235455 bytes) 000462[1] 1767315501.043376 mapread 0x2a8000 thru 0x2b4223 (0xc224 bytes) 000463[1] 1767315501.074339 trunc from 0x776d5e to 0x395942 (0x3e141b bytes) 000467[0] 1767315501.129039 mapwrite 0x3ca000 thru 0x3d8c21 (0xec22 bytes) 000468[0] 1767315501.292797 write 0x545000 thru 0x54506a (0x6b bytes) 000469[1] 1767315501.310060 write_OD 0x1cc000 thru 0x1cefff (0x3000 bytes) 000474[1] 1767315501.435371 mapread 0x0c1000 thru 0x0cec39 (0xdc3a bytes) 000476[0] 1767315501.477512 write 0x6a3000 thru 0x6ac71d (0x971e bytes) 000477[1] 1767315501.528816 mapread 0x3be000 thru 0x3ccfe8 (0xefe9 bytes) 000478[0] 1767315501.590645 read 0x5ee000 thru 0x5f2c51 (0x4c52 bytes) 000479[1] 1767315501.597914 mapread 0x009000 thru 0x00e44b (0x544c bytes) 000480[0] 1767315501.615013 read 0x063000 thru 0x06d41b (0xa41c bytes) 000482[0] 1767315501.623753 write 0x3cc000 thru 0x3cf5c2 (0x35c3 bytes) 000483[0] 1767315501.660716 read 0x5c7000 thru 0x5d251f (0xb520 bytes) 000485[1] 1767315501.686370 mapwrite 0x01c000 thru 0x026a8b (0xaa8c bytes) 000488[0] 1767315501.948304 mapread 0x3d7000 thru 0x3e178a (0xa78b bytes) 000490[1] 1767315502.029354 write_OD 0x514000 thru 0x51dfff (0xa000 bytes) 000494[0] 1767315502.124587 write 0x4e2000 thru 0x4e796c (0x596d bytes) 000495[0] 1767315502.150557 mapwrite 0x6bb000 thru 0x6c0427 (0x5428 bytes) 000496[1] 1767315502.259455 write_OD 0x3d6000 thru 0x3defff (0x9000 bytes) 000497[1] 1767315502.324316 mapread 0x389000 thru 0x3921ab (0x91ac bytes) 000498[1] 1767315502.348933 trunc from 0x6c0428 to 0x57aa8d (0x14599a bytes) 000500[1] 1767315502.507139 mapread 0x0e8000 thru 0x0f7f80 (0xff81 bytes) All operations completed A-OK! PASS 16a (49s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:58:28 (1767315508) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.316644 s, 26.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.467097 s, 18.0 MB/s Chance of close/open is 1 in 50 Seed set to 5511 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x71a108 000003[0] 1767315512.182450 trunc from 00000000 to 0x71a107 (0x71a108 bytes) 000005[1] 1767315512.210671 read 0x469d0d thru 0x46ab21 (0xe15 bytes) 000006[0] 1767315512.275291 mapread 0x13921b thru 0x1393cb (0x1b1 bytes) 000009[0] 1767315512.364317 mapread 0x6d14ad thru 0x6d2794 (0x12e8 bytes) 000011[0] 1767315512.436417 trunc from 0x71a108 to 0x38f428 (0x38acdf bytes) 000012[1] 1767315512.522627 mapread 0x12e1c6 thru 0x12fe62 (0x1c9d bytes) 000015[1] 1767315512.617117 trunc from 0x38f429 to 0x4db8ac (0x14c484 bytes) truncating to largest ever: 0x7959e2 000018[1] 1767315512.763724 trunc from 0x4db8ad to 0x7959e1 (0x2ba135 bytes) 000019[1] 1767315512.792118 trunc from 0x7959e2 to 0x64c080 (0x149961 bytes) 000020[0] 1767315512.813577 mapread 0x1b96e4 thru 0x1ba204 (0xb21 bytes) 000022[1] 1767315512.897891 write 0x6b6efb thru 0x6b8b7f (0x1c85 bytes) 000023[1] 1767315512.915033 read 0x4a4e94 thru 0x4a5023 (0x190 bytes) 000025[1] 1767315512.922324 read 0x1c6c36 thru 0x1c6cd5 (0xa0 bytes) 000026[1] 1767315512.940806 trunc from 0x6b8b80 to 0x3ca234 (0x2ee94b bytes) 000028[1] 1767315513.020974 read 0x2f1e9c thru 0x2f2694 (0x7f9 bytes) 000031[0] 1767315513.203821 trunc from 0x3ca235 to 0x71a6a6 (0x350472 bytes) 000032[0] 1767315513.303304 write 0x0e456d thru 0x0e5fd4 (0x1a68 bytes) 000034[1] 1767315513.338715 trunc from 0x71a6a7 to 0x386c47 (0x393a5f bytes) truncating to largest ever: 0x92f6a6 000035[0] 1767315513.402001 trunc from 0x386c48 to 0x92f6a5 (0x5a8a5e bytes) 000036[1] 1767315513.429772 mapread 0x09cf28 thru 0x09e9b6 (0x1a8f bytes) 000039[0] 1767315513.483932 trunc from 0x92f6a6 to 0x42633b (0x50936a bytes) 000040[1] 1767315513.512873 mapwrite 0x10bcac thru 0x10d861 (0x1bb6 bytes) 000045[0] 1767315513.667800 mapread 0x367784 thru 0x368a98 (0x1315 bytes) 000046[0] 1767315513.733091 write 0x3d46ba thru 0x3d531b (0xc62 bytes) 000048[1] 1767315513.790317 trunc from 0x42633c to 0x090647 (0x395cf4 bytes) 000051[1] 1767315513.888357 write 0x03b728 thru 0x03bc5e (0x537 bytes) 000055[1] 1767315513.985578 write 0x90936b thru 0x909ab0 (0x746 bytes) 000056[0] 1767315514.020868 trunc from 0x909ab1 to 0x1aa6ce (0x75f3e2 bytes) 000060[1] 1767315514.088237 mapwrite 0x60b09c thru 0x60bf8a (0xeef bytes) 000062[1] 1767315514.343402 write 0x9d5a7f thru 0x9d661d (0xb9f bytes) 000063[1] 1767315514.359955 read 0x95ddc3 thru 0x95fa72 (0x1cb0 bytes) 000064[0] 1767315514.364425 trunc from 0x9d661e to 0x072c9f (0x96397e bytes) 000067[1] 1767315514.416350 read 0x0630db thru 0x063c94 (0xbba bytes) 000068[0] 1767315514.442707 write 0x7fc9b1 thru 0x7fd2bc (0x90c bytes) 000069[1] 1767315514.447320 mapwrite 0x061a8e thru 0x062606 (0xb79 bytes) 000073[0] 1767315514.617560 trunc from 0x7fd2bd to 0x705acb (0xf77f1 bytes) 000074[0] 1767315514.682623 mapread 0x0ff7c0 thru 0x100378 (0xbb9 bytes) 000075[1] 1767315514.726047 mapread 0x6b7cd5 thru 0x6b80a3 (0x3cf bytes) 000076[1] 1767315514.776014 mapread 0x06f343 thru 0x070af0 (0x17ae bytes) 000081[0] 1767315514.819890 read 0x3d68ab thru 0x3d7d4f (0x14a5 bytes) 000084[1] 1767315514.826125 trunc from 0x705acc to 0x6d02db (0x357f0 bytes) 000086[1] 1767315514.909865 mapread 0x4b23c2 thru 0x4b24a7 (0xe6 bytes) 000087[1] 1767315514.942990 mapwrite 0x3c8f0d thru 0x3ca71c (0x1810 bytes) 000090[1] 1767315515.000882 mapwrite 0x4fc8ae thru 0x4fe40a (0x1b5d bytes) 000092[1] 1767315515.123261 read 0x153600 thru 0x1549f5 (0x13f6 bytes) 000093[0] 1767315515.164992 write 0x5266f8 thru 0x5274cd (0xdd6 bytes) truncating to largest ever: 0x9851d0 000094[1] 1767315515.233379 trunc from 0x6d02dc to 0x9851cf (0x2b4ef4 bytes) 000095[1] 1767315515.317555 mapread 0x2c9904 thru 0x2ca8bf (0xfbc bytes) 000096[1] 1767315515.335311 mapwrite 0x49da56 thru 0x49e270 (0x81b bytes) 000097[0] 1767315515.420780 read 0x851523 thru 0x852efe (0x19dc bytes) 000098[1] 1767315515.458501 mapread 0x0a8195 thru 0x0a9255 (0x10c1 bytes) 000099[1] 1767315515.503874 mapread 0x933e4a thru 0x935bc0 (0x1d77 bytes) 000101[1] 1767315515.533567 trunc from 0x9851d0 to 0x75d7eb (0x2279e4 bytes) 000102[0] 1767315515.566147 read 0x58b169 thru 0x58be66 (0xcfe bytes) 000103[1] 1767315515.603722 write 0x02474e thru 0x0265d8 (0x1e8b bytes) 000105[0] 1767315515.658343 write 0x359e4c thru 0x35bc35 (0x1dea bytes) 000107[0] 1767315515.711986 read 0x6627ea thru 0x663b14 (0x132b bytes) 000108[0] 1767315515.719720 read 0x6cb25e thru 0x6cce58 (0x1bfb bytes) 000109[0] 1767315515.758392 read 0x4605b3 thru 0x460c29 (0x677 bytes) 000110[1] 1767315515.768792 mapread 0x1fe3ff thru 0x1ffc83 (0x1885 bytes) truncating to largest ever: 0x9a683b 000111[1] 1767315515.815450 trunc from 0x75d7ec to 0x9a683a (0x24904f bytes) 000116[0] 1767315515.914402 mapwrite 0x37a66b thru 0x37a849 (0x1df bytes) 000117[1] 1767315516.049098 trunc from 0x9a683b to 0x3da5d1 (0x5cc269 bytes) 000118[0] 1767315516.095404 write 0x480959 thru 0x482129 (0x17d1 bytes) 000119[1] 1767315516.153837 read 0x235f94 thru 0x236664 (0x6d1 bytes) 000120[1] 1767315516.265791 read 0x36a9c8 thru 0x36bf33 (0x156c bytes) 000123[0] 1767315516.310145 trunc from 0x48212a to 0x929c74 (0x4a7b4b bytes) 000124[1] 1767315516.376629 write 0x1656db thru 0x166075 (0x99b bytes) 000125[1] 1767315516.540398 write 0x290f60 thru 0x2914d9 (0x57a bytes) 000127[0] 1767315516.585366 write 0x33d043 thru 0x33d7a8 (0x766 bytes) 000128[1] 1767315516.685550 mapwrite 0x0c955d thru 0x0ca25a (0xcfe bytes) 000129[0] 1767315516.817684 trunc from 0x929c75 to 0x573d38 (0x3b5f3c bytes) 000133[1] 1767315516.879996 mapwrite 0x5539f4 thru 0x555465 (0x1a72 bytes) 000137[0] 1767315517.094945 read 0x1a1f89 thru 0x1a2654 (0x6cc bytes) 000139[1] 1767315517.144840 mapread 0x519218 thru 0x519b17 (0x900 bytes) 000140[1] 1767315517.176912 write 0x36772b thru 0x369304 (0x1bda bytes) 000141[0] 1767315517.248855 read 0x448e2a thru 0x44a9d7 (0x1bae bytes) 000142[1] 1767315517.279446 read 0x11bc27 thru 0x11d3df (0x17b9 bytes) 000143[0] 1767315517.304296 read 0x552395 thru 0x55432c (0x1f98 bytes) 000147[1] 1767315517.344335 write 0x32e93b thru 0x32fa49 (0x110f bytes) 000151[0] 1767315517.388954 read 0x47cf9d thru 0x47e3a5 (0x1409 bytes) 000152[1] 1767315517.394694 mapread 0x1b8f37 thru 0x1b9427 (0x4f1 bytes) 000153[1] 1767315517.404642 mapwrite 0x4043bb thru 0x406181 (0x1dc7 bytes) 000155[1] 1767315517.486453 mapwrite 0x72b546 thru 0x72c4b8 (0xf73 bytes) 000156[0] 1767315517.586635 mapwrite 0x33ca22 thru 0x33d677 (0xc56 bytes) 000157[0] 1767315517.685348 read 0x252674 thru 0x253d58 (0x16e5 bytes) 000158[1] 1767315517.704132 write 0x7381ab thru 0x738d95 (0xbeb bytes) 000159[1] 1767315517.723740 trunc from 0x738d96 to 0x19e805 (0x59a590 bytes) 000161[0] 1767315517.788349 mapwrite 0x526a08 thru 0x527b9a (0x1193 bytes) 000163[0] 1767315517.918965 mapwrite 0x863f60 thru 0x865812 (0x18b3 bytes) 000164[0] 1767315517.994625 mapwrite 0x9e2ba7 thru 0x9e3e64 (0x12be bytes) 000166[0] 1767315518.050618 write 0x464e94 thru 0x466dce (0x1f3b bytes) 000172[1] 1767315518.101213 mapwrite 0x28ab7d thru 0x28ad9f (0x223 bytes) 000174[0] 1767315518.195960 trunc from 0x9e3e65 to 0x3c91d3 (0x61ac91 bytes) 000175[1] 1767315518.245782 write 0x5d5f3d thru 0x5d6b2a (0xbee bytes) 000177[0] 1767315518.300563 mapwrite 0x946046 thru 0x947fad (0x1f68 bytes) 000180[0] 1767315518.437430 mapwrite 0x590816 thru 0x5920c9 (0x18b4 bytes) 000182[0] 1767315518.541960 mapread 0x75af02 thru 0x75b481 (0x580 bytes) 000186[0] 1767315518.555908 trunc from 0x947fae to 0x9974bf (0x4f512 bytes) 000188[1] 1767315518.608702 mapread 0x21046a thru 0x210a9a (0x631 bytes) 000195[0] 1767315518.687651 mapread 0x22a1a2 thru 0x22c00c (0x1e6b bytes) 000199[0] 1767315518.721707 write 0x195bbc thru 0x19604d (0x492 bytes) 000201[0] 1767315518.812930 mapread 0x4a06e8 thru 0x4a1eeb (0x1804 bytes) 000202[0] 1767315518.841744 trunc from 0x9974c0 to 0x2e498e (0x6b2b31 bytes) 000203[1] 1767315518.896476 trunc from 0x2e498f to 0x998698 (0x6b3d0a bytes) 000204[0] 1767315518.934119 trunc from 0x998699 to 0x27a446 (0x71e252 bytes) 000205[1] 1767315518.973903 read 0x1a381d thru 0x1a41ac (0x990 bytes) 000206[1] 1767315518.991870 trunc from 0x27a447 to 0x002715 (0x277d31 bytes) 000208[1] 1767315519.065281 mapwrite 0x7a3fe2 thru 0x7a5561 (0x1580 bytes) 000209[1] 1767315519.170103 read 0x4beee3 thru 0x4c0338 (0x1456 bytes) 000211[1] 1767315519.176883 write 0x92d67f thru 0x92ea72 (0x13f4 bytes) 000212[0] 1767315519.178784 mapwrite 0x8b2491 thru 0x8b3f55 (0x1ac5 bytes) 000214[0] 1767315519.319071 mapwrite 0x939d47 thru 0x93a631 (0x8eb bytes) 000215[0] 1767315519.473549 write 0x48fa61 thru 0x4917ef (0x1d8f bytes) 000218[1] 1767315519.596112 write 0x007a79 thru 0x008d23 (0x12ab bytes) 000219[1] 1767315519.632985 write 0x4a7f9d thru 0x4a87aa (0x80e bytes) 000222[0] 1767315519.685544 write 0x1c3de4 thru 0x1c4f65 (0x1182 bytes) 000223[1] 1767315519.718690 mapread 0x1160a9 thru 0x117158 (0x10b0 bytes) 000226[1] 1767315519.749330 mapwrite 0x2b0872 thru 0x2b1a5c (0x11eb bytes) 000227[1] 1767315519.810223 write 0x68b214 thru 0x68bf29 (0xd16 bytes) 000229[1] 1767315519.817565 mapread 0x14c771 thru 0x14d67f (0xf0f bytes) 000236[1] 1767315519.911323 write 0x21bca8 thru 0x21c4e7 (0x840 bytes) 000239[1] 1767315519.982550 write 0x29369d thru 0x2954bb (0x1e1f bytes) 000240[1] 1767315520.023903 read 0x5dda9c thru 0x5df4bf (0x1a24 bytes) 000241[1] 1767315520.030636 mapread 0x7111b8 thru 0x711c0c (0xa55 bytes) 000242[1] 1767315520.039912 mapwrite 0x5937c0 thru 0x594735 (0xf76 bytes) 000244[0] 1767315520.120594 write 0x15766b thru 0x157caf (0x645 bytes) 000245[1] 1767315520.160749 write 0x380fd5 thru 0x382949 (0x1975 bytes) 000246[1] 1767315520.215359 write 0x5c0a86 thru 0x5c20e8 (0x1663 bytes) 000247[0] 1767315520.249732 trunc from 0x93a632 to 0x25597d (0x6e4cb4 bytes) 000249[0] 1767315520.325689 mapread 0x19c591 thru 0x19cd0d (0x77d bytes) 000250[0] 1767315520.363005 mapread 0x0b9347 thru 0x0ba6a0 (0x135a bytes) 000251[0] 1767315520.417061 mapread 0x010c9f thru 0x0119f4 (0xd56 bytes) 000253[0] 1767315520.492058 trunc from 0x25597e to 0x17ba95 (0xd9ee8 bytes) 000254[1] 1767315520.552663 trunc from 0x17ba96 to 0x193ff3 (0x1855e bytes) 000255[0] 1767315520.703409 write 0x1e8042 thru 0x1e8a10 (0x9cf bytes) 000256[0] 1767315520.795104 mapread 0x14187b thru 0x142e72 (0x15f8 bytes) 000257[0] 1767315520.865122 mapread 0x15f2cf thru 0x15f9cf (0x701 bytes) 000258[0] 1767315520.919585 trunc from 0x1e8a11 to 0x4d9729 (0x2f0d19 bytes) 000259[1] 1767315520.985972 mapwrite 0x3d0051 thru 0x3d129e (0x124e bytes) 000262[0] 1767315521.180531 mapread 0x00e441 thru 0x00edf3 (0x9b3 bytes) 000263[1] 1767315521.215628 mapread 0x3059d4 thru 0x3072b8 (0x18e5 bytes) 000269[0] 1767315521.299787 mapread 0x1068c5 thru 0x1070d7 (0x813 bytes) 000271[0] 1767315521.324721 mapwrite 0x7726c7 thru 0x7729be (0x2f8 bytes) 000273[0] 1767315521.450125 mapwrite 0x12e5fc thru 0x12f4eb (0xef0 bytes) 000278[0] 1767315521.515081 read 0x2a562f thru 0x2a7442 (0x1e14 bytes) 000284[1] 1767315521.587165 write 0x62d603 thru 0x62e330 (0xd2e bytes) 000286[0] 1767315521.638147 mapwrite 0x869d9d thru 0x86b70a (0x196e bytes) 000288[1] 1767315521.808693 read 0x73360f thru 0x733da2 (0x794 bytes) 000289[0] 1767315521.845651 mapwrite 0x081797 thru 0x082561 (0xdcb bytes) 000290[1] 1767315521.906476 trunc from 0x86b70b to 0x58a4af (0x2e125b bytes) 000293[1] 1767315521.973261 write 0x97b6dc thru 0x97b6ee (0x13 bytes) 000295[0] 1767315521.995234 mapread 0x4bc6b9 thru 0x4bc9e9 (0x331 bytes) 000296[0] 1767315522.030589 read 0x76b7a5 thru 0x76c02f (0x88b bytes) 000300[0] 1767315522.102541 trunc from 0x97b6ef to 0x26b818 (0x70fed6 bytes) 000302[1] 1767315522.181649 trunc from 0x26b819 to 0x34fe2d (0xe4615 bytes) 000303[1] 1767315522.220617 read 0x1bea35 thru 0x1bf11e (0x6ea bytes) 000304[0] 1767315522.298515 mapwrite 0x4fbf9c thru 0x4fdafd (0x1b62 bytes) 000306[1] 1767315522.458002 mapwrite 0x767126 thru 0x768d27 (0x1c02 bytes) 000312[0] 1767315522.710121 write 0x6a561f thru 0x6a5e42 (0x824 bytes) 000313[1] 1767315522.742501 read 0x04d883 thru 0x04e00d (0x78b bytes) 000314[0] 1767315522.757515 mapwrite 0x6e7530 thru 0x6e8065 (0xb36 bytes) 000316[1] 1767315522.839910 mapwrite 0x79d2c3 thru 0x79f049 (0x1d87 bytes) 000318[0] 1767315522.979666 read 0x4f8cef thru 0x4fa1d0 (0x14e2 bytes) 000320[0] 1767315523.043176 write 0x9e8fdb thru 0x9e9e37 (0xe5d bytes) 000325[0] 1767315523.112092 read 0x8858bb thru 0x886e0e (0x1554 bytes) 000327[1] 1767315523.137698 mapread 0x25f748 thru 0x26020a (0xac3 bytes) 000328[0] 1767315523.198803 mapwrite 0x9bac5e thru 0x9bb5a8 (0x94b bytes) 000333[1] 1767315523.289951 mapwrite 0x84d46d thru 0x84e8fa (0x148e bytes) 000338[0] 1767315523.433476 read 0x7badfc thru 0x7bbcd4 (0xed9 bytes) 000339[0] 1767315523.482758 mapread 0x3d5dea thru 0x3d6eab (0x10c2 bytes) 000340[0] 1767315523.515440 mapwrite 0x13dd65 thru 0x13e3b4 (0x650 bytes) 000342[0] 1767315523.602461 write 0x52247a thru 0x523e1c (0x19a3 bytes) 000345[0] 1767315523.662413 read 0x61ecaf thru 0x61eda7 (0xf9 bytes) 000348[0] 1767315523.748687 read 0x616ac8 thru 0x6176ad (0xbe6 bytes) 000349[1] 1767315523.783282 read 0x3abc1d thru 0x3adad9 (0x1ebd bytes) 000351[1] 1767315523.842260 read 0x70dffb thru 0x70e440 (0x446 bytes) 000354[0] 1767315523.876999 trunc from 0x9e9e38 to 0x87c02b (0x16de0c bytes) 000355[1] 1767315523.948209 write 0x0a902f thru 0x0a90cc (0x9e bytes) 000357[1] 1767315523.987630 read 0x48c01f thru 0x48dc0a (0x1bec bytes) 000358[0] 1767315524.026625 mapread 0x319174 thru 0x319d89 (0xc16 bytes) 000360[1] 1767315524.078098 read 0x4baf14 thru 0x4bba7f (0xb6c bytes) 000364[0] 1767315524.087871 mapread 0x8238f3 thru 0x824d5e (0x146c bytes) 000366[0] 1767315524.098253 read 0x416da2 thru 0x4174ab (0x70a bytes) 000367[0] 1767315524.109266 mapwrite 0x1561b3 thru 0x156c33 (0xa81 bytes) 000369[1] 1767315524.224594 mapread 0x34200c thru 0x343bed (0x1be2 bytes) 000375[0] 1767315524.257588 write 0x657063 thru 0x658ffd (0x1f9b bytes) 000381[0] 1767315524.306327 write 0x3412d2 thru 0x3425f2 (0x1321 bytes) truncating to largest ever: 0x9dddcc 000384[0] 1767315524.355549 trunc from 0x87c02c to 0x9dddcb (0x161da0 bytes) 000387[0] 1767315524.446172 read 0x60be72 thru 0x60cdcb (0xf5a bytes) 000389[1] 1767315524.456168 write 0x4942cd thru 0x4958cc (0x1600 bytes) 000393[1] 1767315524.642895 mapread 0x4c183e thru 0x4c29d2 (0x1195 bytes) 000394[0] 1767315524.695036 mapread 0x52becf thru 0x52c94e (0xa80 bytes) 000398[1] 1767315524.751695 read 0x27fefc thru 0x28088f (0x994 bytes) 000403[1] 1767315524.792052 write 0x470795 thru 0x471fdf (0x184b bytes) 000404[1] 1767315524.882248 mapread 0x77be5a thru 0x77dd79 (0x1f20 bytes) 000405[0] 1767315524.918796 mapwrite 0x27e869 thru 0x27f0a6 (0x83e bytes) 000406[0] 1767315525.066217 trunc from 0x9dddcc to 0x94bc3f (0x9218c bytes) 000407[1] 1767315525.116579 read 0x0f886e thru 0x0f9456 (0xbe9 bytes) 000408[1] 1767315525.147488 read 0x23c816 thru 0x23e789 (0x1f74 bytes) 000410[0] 1767315525.167964 write 0x4627f8 thru 0x463199 (0x9a2 bytes) 000414[1] 1767315525.200639 mapwrite 0x98eb8f thru 0x99084e (0x1cc0 bytes) 000415[0] 1767315525.376015 read 0x731da3 thru 0x73211f (0x37d bytes) 000417[1] 1767315525.436358 write 0x67ce07 thru 0x67e214 (0x140e bytes) 000418[1] 1767315525.500744 write 0x1a0ad9 thru 0x1a2153 (0x167b bytes) 000419[1] 1767315525.509674 mapread 0x39bf70 thru 0x39cde3 (0xe74 bytes) 000424[1] 1767315525.565909 mapread 0x1ade78 thru 0x1adf04 (0x8d bytes) 000425[1] 1767315525.571458 mapwrite 0x0a78fa thru 0x0a8ea6 (0x15ad bytes) 000426[0] 1767315525.635034 mapwrite 0x5361d0 thru 0x537fc6 (0x1df7 bytes) 000428[1] 1767315525.749298 write 0x162f49 thru 0x164893 (0x194b bytes) 000429[1] 1767315525.821752 read 0x004ca5 thru 0x005943 (0xc9f bytes) 000430[0] 1767315525.856464 mapwrite 0x810661 thru 0x811ea6 (0x1846 bytes) 000433[0] 1767315526.057681 write 0x1f4b83 thru 0x1f557d (0x9fb bytes) 000434[0] 1767315526.075172 trunc from 0x99084f to 0x72d1e6 (0x263668 bytes) 000436[1] 1767315526.139514 mapread 0x1deb19 thru 0x1df7a3 (0xc8b bytes) 000437[0] 1767315526.186752 read 0x5dcb08 thru 0x5de410 (0x1909 bytes) 000442[0] 1767315526.225726 write 0x640222 thru 0x641b9b (0x197a bytes) 000443[1] 1767315526.236622 read 0x0c5d13 thru 0x0c72fe (0x15ec bytes) 000445[1] 1767315526.256767 read 0x3f6248 thru 0x3f699c (0x755 bytes) 000446[0] 1767315526.262716 mapwrite 0x48e6bd thru 0x48f01d (0x961 bytes) 000447[1] 1767315526.370548 write 0x279d39 thru 0x27b0d4 (0x139c bytes) 000448[1] 1767315526.422704 read 0x0d5c0a thru 0x0d686f (0xc66 bytes) 000449[0] 1767315526.440756 read 0x6bd9fe thru 0x6be716 (0xd19 bytes) 000450[0] 1767315526.480306 trunc from 0x72d1e7 to 0x826626 (0xf9440 bytes) 000453[1] 1767315526.566274 read 0x474787 thru 0x474789 (0x3 bytes) 000455[1] 1767315526.607972 read 0x25bf90 thru 0x25c323 (0x394 bytes) 000458[1] 1767315526.623359 trunc from 0x826627 to 0x094b37 (0x791aef bytes) 000460[0] 1767315526.705311 read 0x089013 thru 0x08aa78 (0x1a66 bytes) 000461[0] 1767315526.783750 mapwrite 0x1d10e3 thru 0x1d24dd (0x13fb bytes) 000462[1] 1767315527.037075 write 0x6fe9ab thru 0x700610 (0x1c66 bytes) 000463[1] 1767315527.081314 write 0x08ec27 thru 0x09067b (0x1a55 bytes) 000464[1] 1767315527.170157 write 0x4a732b thru 0x4a8c5b (0x1931 bytes) 000465[1] 1767315527.251108 write 0x3e4efe thru 0x3e5d38 (0xe3b bytes) 000467[0] 1767315527.267779 mapread 0x655484 thru 0x6560fd (0xc7a bytes) 000468[0] 1767315527.385320 trunc from 0x700611 to 0x5b52f2 (0x14b31e bytes) 000469[1] 1767315527.483130 write 0x64a9c5 thru 0x64b5b0 (0xbec bytes) 000473[0] 1767315527.571066 write 0x14858f thru 0x14a093 (0x1b05 bytes) 000474[0] 1767315527.648573 trunc from 0x64b5b1 to 0x09616d (0x5b5443 bytes) 000477[1] 1767315527.913805 mapread 0x063cac thru 0x065956 (0x1cab bytes) 000478[1] 1767315527.989759 write 0x16ea76 thru 0x1706e7 (0x1c72 bytes) 000479[0] 1767315528.057115 mapread 0x0027e8 thru 0x003465 (0xc7e bytes) 000480[1] 1767315528.130174 trunc from 0x1706e8 to 0x8fffd4 (0x78f8ed bytes) 000481[1] 1767315528.216003 mapread 0x723056 thru 0x724405 (0x13b0 bytes) 000482[0] 1767315528.228175 trunc from 0x8fffd5 to 0x58413c (0x37be98 bytes) 000486[1] 1767315528.384000 trunc from 0x58413d to 0x804922 (0x2807e6 bytes) 000488[1] 1767315528.509105 read 0x316c58 thru 0x31800d (0x13b6 bytes) 000489[0] 1767315528.535318 read 0x5ea296 thru 0x5ea9be (0x729 bytes) 000490[1] 1767315528.563509 write 0x4a9eb1 thru 0x4aa883 (0x9d3 bytes) 000491[0] 1767315528.581001 mapwrite 0x733296 thru 0x73480e (0x1579 bytes) 000492[1] 1767315528.714025 trunc from 0x804923 to 0x401cb1 (0x402c71 bytes) 000494[1] 1767315528.888291 trunc from 0x401cb2 to 0x7a4f67 (0x3a32b6 bytes) 000495[0] 1767315528.967333 trunc from 0x7a4f68 to 0x2d2959 (0x4d260e bytes) 000497[0] 1767315529.074003 mapread 0x0e0d12 thru 0x0e21f8 (0x14e7 bytes) 000498[0] 1767315529.117996 mapread 0x22beb9 thru 0x22cf3d (0x1085 bytes) 000499[0] 1767315529.171809 trunc from 0x2d295a to 0x061846 (0x271113 bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:58:55 (1767315535) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:58:59 (1767315539) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.521274 s, 16.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.339883 s, 24.7 MB/s ldlm.namespaces.MGC192.168.203.117@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d57c2bba800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d57d015c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d57d015c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d57d015c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2073 s, 9.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.59502 s, 40.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.33551 s, 31.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.33246 s, 16.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.46438 s, 12.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.21628 s, 24.9 MB/s PASS 16d (50s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:59:49 (1767315589) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.347288 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113389 s, 9.2 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:59:57 (1767315597) TIMEOUT PASS 16f (32s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 20:00:29 (1767315629) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 20:01:00 (1767315660) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 20:01:08 (1767315668) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 20:01:17 (1767315677) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89797 s, 9.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.47246 s, 17.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.92669 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.65397 s, 15.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12292 s, 12.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3393 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95683 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.85151 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.02968 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.96876 s, 6.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.1467 s, 6.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54719 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79167 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19046 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17698 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.83146 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.5692 s, 5.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.11694 s, 5.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.14528 s, 5.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (45s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 20:02:02 (1767315722) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Thu Jan 1 20:02:56 EST 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu Jan 1 20:02:56 EST 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Thu Jan 1 20:02:57 EST 2026 + fsx 4 done: rc=0 Thu Jan 1 20:02:57 EST 2026 + fsx 5 done: rc=0 Thu Jan 1 20:02:57 EST 2026 PASS 16k (65s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 20:03:07 (1767315787) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 20:03:18 (1767315798) mmap test1: basic mmap operation (PASS, 0.11855s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10743s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.3697s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.67s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11815s) mmap test6: check mmap write/read content on two nodes (PASS, 0.25539s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2136s) mmap test11: multi-thread shared mmap (PASS, 1.0145s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1209s) PASS 18 (23s) == sanityn test 19: test concurrent uncached read races ========================================================================= 20:03:41 (1767315821) oleg317-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 20:03:46 (1767315826) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 20:03:53 (1767315833) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 20:04:00 (1767315840) now is 1767315843 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 new atime is 1767315904 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 20:05:10 (1767315910) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117198 398 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118066 402 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117198 398 116800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118066 402 117664 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117198 398 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118066 402 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 20:05:20 (1767315920) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 20:05:26 (1767315926) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 20:05:36 (1767315936) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 20:05:39 (1767315939) utime: good mknod times 1767315940 <= 1767315941 <= 1767315941 for /mnt/lustre/f26a utime: good mknod times 1767315940 <= 1767315941 <= 1767315941 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 20:05:46 (1767315946) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 20:05:56 (1767315956) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-01 20:05:58.000000000 -0500 Birth: 2026-01-01 20:05:58.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-01 20:05:58.000000000 -0500 Birth: 2026-01-01 20:05:58.000000000 -0500 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 20:06:06 (1767315966) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 20:06:15 (1767315975) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 20:06:23 (1767315983) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg317-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.117@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 20:06:45 (1767316005) -rw-r--r-- 1 root root 0 Jan 1 20:06 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 20:06:55 (1767316015) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 44917 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 20:07:03 (1767316023) PASS 31t (9s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 20:07:13 (1767316033) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 20:07:17 (1767316037) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 20:07:20 (1767316040) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 20:07:24 (1767316044) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 20:07:27 (1767316047) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57c2bba800.ost_server_uuid in IDLE state after 0 sec PASS 34 (138s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 20:09:45 (1767316185) fail_loc=0x80000317 Race attempt 0 Wait for 50968 51002 for 60 sec... - open/close 925 (time 1767316202.34 total 10.00 last 92.49) - open/close 1860 (time 1767316212.34 total 20.01 last 93.44) - open/close 2576 (time 1767316222.35 total 30.01 last 71.56) - open/close 3435 (time 1767316232.35 total 40.02 last 85.86) total: 4000 open/close in 47.73 seconds: 83.81 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.117@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.117@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 20:10:58 (1767316258) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg317-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61299 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.73784 s, 30.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39407 s, 37.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41365 s, 37.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13378 s, 46.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2595 s, 41.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.67245 s, 31.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65627 s, 31.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42743 s, 36.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57262 s, 33.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81346 s, 28.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7441408) after(7492608) PASS 36 (304s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 20:16:02 (1767316562) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 - create 1439 (time 1767316576.17 total 10.00 last 143.85) - create 3024 (time 1767316586.17 total 20.01 last 158.43) - create 4465 (time 1767316596.17 total 30.01 last 144.09) - create 6053 (time 1767316606.18 total 40.02 last 158.69) - create 7648 (time 1767316616.19 total 50.03 last 159.28) - create 9085 (time 1767316626.19 total 60.03 last 143.68) total: 10000 create in 66.09 seconds: 151.31 ops/second PASS 37 (77s) == sanityn test 39a: file mtime does not change after rename ========================================================== 20:17:20 (1767316640) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 20:17:28 (1767316648) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 20:17:37 (1767316657) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 20:17:45 (1767316665) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 20:17:53 (1767316673) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 20:18:11 (1767316691) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 20:18:29 (1767316709) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 20:18:46 (1767316726) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 20:19:04 (1767316744) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 20:19:17 (1767316757) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 20:19:31 (1767316771) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 20:19:46 (1767316786) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 20:19:59 (1767316799) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 20:20:13 (1767316813) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 20:20:26 (1767316826) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 20:20:38 (1767316838) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 20:20:51 (1767316851) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1056s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:38:27 (1767317907) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:38:40 (1767317920) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:38:53 (1767317933) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:39:07 (1767317947) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:39:20 (1767317960) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (12s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:39:33 (1767317973) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:39:45 (1767317985) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:39:58 (1767317998) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:40:11 (1767318011) PASS 43a (62s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:41:13 (1767318073) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:41:27 (1767318087) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:41:40 (1767318100) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:41:52 (1767318112) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:42:05 (1767318125) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:42:18 (1767318138) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:42:31 (1767318151) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:42:46 (1767318166) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:42:49 (1767318169) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (127s) == sanityn test 43k: unlink vs create ==================== 20:44:56 (1767318296) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1198s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 21:04:55 (1767319495) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jan 1 21:05:04 EST 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 21:05:09 (1767319509) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 21:05:23 (1767319523) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 21:05:37 (1767319537) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 21:05:51 (1767319551) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 21:06:07 (1767319567) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 21:06:22 (1767319582) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 21:06:35 (1767319595) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 21:06:50 (1767319610) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 21:06:53 (1767319613) PASS 45a (112s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 21:08:45 (1767319725) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 21:08:59 (1767319739) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 21:09:12 (1767319752) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 21:09:26 (1767319766) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 21:09:40 (1767319780) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 21:09:53 (1767319793) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 21:10:08 (1767319808) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 21:10:23 (1767319823) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 21:10:27 (1767319827) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1167s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 21:29:53 (1767320993) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 21:30:05 (1767321005) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 21:30:18 (1767321018) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 21:30:32 (1767321032) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 21:30:46 (1767321046) Conflict PASS 46e (11s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 21:30:58 (1767321058) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 21:31:10 (1767321070) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 21:31:23 (1767321083) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 21:31:38 (1767321098) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 21:31:42 (1767321102) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 21:31:45 (1767321105) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 21:31:48 (1767321108) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 21:31:52 (1767321112) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 21:31:55 (1767321115) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 21:31:58 (1767321118) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 21:32:01 (1767321121) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 21:32:04 (1767321124) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0918947 s, 111 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:32:16 (1767321136) 0+1 records in 0+1 records out 158 bytes copied, 0.0294191 s, 5.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (218158) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 21:32:25 (1767321145) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0129975 s, 78.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377168 s, 27.8 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:32:43 (1767321163) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0249857 s, 41.0 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:32:55 (1767321175) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11374 s, 9.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 21:33:03 (1767321183) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 21:33:11 (1767321191) llite.lustre-ffff8d57c2bba800.xattr_cache=1 llite.lustre-ffff8d57d015c000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:12.000000000 -0500 Modify: 2026-01-01 21:33:12.000000000 -0500 Change: 2026-01-01 21:33:12.000000000 -0500 Birth: 2026-01-01 21:33:13.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:19.000000000 -0500 Modify: 2026-01-01 21:33:20.000000000 -0500 Change: 2026-01-01 21:33:20.000000000 -0500 Birth: 2026-01-01 21:33:19.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:25.000000000 -0500 Modify: 2026-01-01 21:33:25.000000000 -0500 Change: 2026-01-01 21:33:25.000000000 -0500 Birth: 2026-01-01 21:33:25.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:25.000000000 -0500 Modify: 2026-01-01 21:33:26.000000000 -0500 Change: 2026-01-01 21:33:26.000000000 -0500 Birth: 2026-01-01 21:33:25.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:31.000000000 -0500 Modify: 2026-01-01 21:33:32.000000000 -0500 Change: 2026-01-01 21:33:32.000000000 -0500 Birth: 2026-01-01 21:33:31.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:33:31.000000000 -0500 Modify: 2026-01-01 21:33:32.000000000 -0500 Change: 2026-01-01 21:33:32.000000000 -0500 Birth: 2026-01-01 21:33:31.000000000 -0500 PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 21:33:42 (1767321222) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 21:33:54 (1767321234) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 21:34:08 (1767321248) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 21:34:25 (1767321265) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 21:34:41 (1767321281) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 21:34:44 (1767321284) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 21:35:03 (1767321303) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 21:35:25 (1767321325) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1767321327 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.73541 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (90s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 21:36:55 (1767321415) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1767321419 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.14088 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (118s) == sanityn test 60: Verify data_version behaviour ======== 21:38:54 (1767321534) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 21:39:01 (1767321541) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 21:39:08 (1767321548) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 21:39:16 (1767321556) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0123805 s, 5.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 21:39:19 (1767321559) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0156497 s, 4.2 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 21:39:23 (1767321563) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 21:39:25 (1767321565) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0278347 s, 2.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 21:39:28 (1767321568) llite.lustre-ffff8d57c2bba800.xattr_cache=1 llite.lustre-ffff8d57d015c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 21:39:34 (1767321574) llite.lustre-ffff8d57c2bba800.xattr_cache=1 llite.lustre-ffff8d57d015c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 21:39:41 (1767321581) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0150952 s, 67.8 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0234487 s, 43.7 kB/s 230962: get lock2 230963: get lock1 230963: done 230963: get lock2 230962: done 230962: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 230962: cannot get lock1: Resource deadlock avoided 230963: done 230963: exit rc=0 230962: done 230962: exit rc=0 PASS 74 (10s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 21:39:51 (1767321591) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0827546 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315029 s, 33.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 231545 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 21:40:03 (1767321603) opening files: ........ closing files: ........ PASS 76 (66s) == sanityn test 77a: check FIFO NRS policy =============== 21:41:09 (1767321669) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473425 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721544 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702106 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136784 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841231 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184975 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178092 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269154 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224778 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307063 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263887 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328758 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320573 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300006 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271369 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267333 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365226 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033567 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165874 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322624 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07607 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144129 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00810318 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123567 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160635 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512254 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626355 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398141 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237545 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296012 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170128 s, 61.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231036 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125361 s, 83.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 21:41:18 (1767321678) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.37078 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859239 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921816 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766782 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891334 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184711 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169246 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217476 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163342 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288925 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237213 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254262 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308068 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323318 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343892 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249728 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304733 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250463 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249323 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054664 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668623 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901682 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584425 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772807 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116717 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563946 s, 18.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458211 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439466 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443738 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04029 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336237 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431045 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394363 s, 26.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.380932 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717441 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793378 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145276 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162206 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220587 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215528 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267728 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288404 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230286 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316147 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370073 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428038 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352067 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411673 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411203 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45328 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244118 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321819 s, 326 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347461 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607384 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658821 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040837 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668821 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418429 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432477 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054134 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103619 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295165 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155079 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530201 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536156 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812992 s, 12.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 21:41:30 (1767321690) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445094 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037346 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520305 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12216 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839434 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120531 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161751 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142015 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163146 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114456 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145932 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155256 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159106 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174637 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236206 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217971 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237855 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059822 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0064456 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511228 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527829 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701502 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341094 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401267 s, 261 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321824 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453065 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343764 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351537 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623403 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461784 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422294 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568314 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283715 s, 37.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.41866 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523499 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083914 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145992 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185079 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158182 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222882 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227924 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266695 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249824 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30053 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25405 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329101 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308614 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276265 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31072 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372053 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401871 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464494 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00326381 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032019 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286846 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745599 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461951 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422171 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319995 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365814 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333228 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353057 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398714 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367149 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660204 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505051 s, 20.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.506529 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512934 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655693 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104564 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125237 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201808 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150153 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198663 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239898 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242014 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207851 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225834 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263856 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209399 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257925 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289441 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277967 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839372 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406753 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386099 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550428 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237603 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544346 s, 193 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161533 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267692 s, 3.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154196 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115754 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104372 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990369 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953271 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232472 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820754 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750803 s, 14.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 21:41:45 (1767321705) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.490247 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882825 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138719 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121035 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135704 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933549 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106763 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180391 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141073 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180222 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188521 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226949 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292519 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244486 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187762 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30332 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374417 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347994 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429424 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00264026 s, 397 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138268 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095671 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809588 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147355 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119029 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153686 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633132 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654182 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559776 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446162 s, 23.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422012 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443789 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394129 s, 26.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473387 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503251 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929891 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111743 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909871 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112369 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246657 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130481 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141506 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162922 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241806 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25862 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21926 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25781 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215767 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284225 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27628 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569207 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564051 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886835 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037759 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415106 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782079 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450195 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493473 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577782 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117911 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606758 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638355 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684459 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686524 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717751 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108917 s, 9.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371068 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353515 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725195 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585826 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072161 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119205 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141784 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094943 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177936 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08337 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165044 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183733 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268509 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154226 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230529 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198475 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210787 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136114 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329592 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110028 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587993 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0025196 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108587 s, 96.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323267 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00743007 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248966 s, 421 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120004 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131779 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078419 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803496 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10702 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323539 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332069 s, 31.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (14s) == sanityn test 77e: check TBF NID nrs policy ============ 21:42:00 (1767321720) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.17@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.475884 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797745 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112728 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112156 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251085 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287291 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189615 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303004 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33509 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322964 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340887 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330902 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258232 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276575 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314972 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306605 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414895 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976341 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885019 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904178 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116461 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090645 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100188 s, 10.5 MB/s1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104504 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183116 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137043 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177931 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142872 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154948 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139448 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208443 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149014 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152218 s, 6.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449563 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727022 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992521 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694723 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155105 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205308 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147549 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274718 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188657 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288512 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18597 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195316 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242077 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187549 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294122 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242619 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216836 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629372 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00937352 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791034 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265999 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813389 s, 129 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957061 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107061 s, 97.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607061 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805026 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603028 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677291 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522661 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521676 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826383 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469183 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026371 s, 39.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.374265 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506894 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997149 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107148 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148867 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113922 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929827 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274315 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167851 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239306 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229931 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358582 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297617 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281088 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3593 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323988 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379394 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516261 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897837 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817668 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311057 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265821 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383088 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037352 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396889 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475424 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394236 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512849 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365901 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438729 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465412 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684465 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065609 s, 16.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.495957 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958847 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839114 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132102 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123226 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18948 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208597 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22793 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22008 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266048 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278962 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322013 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492634 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38388 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450897 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417311 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461417 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125429 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00530516 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00634783 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886902 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260655 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322463 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874331 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023326 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116131 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320395 s, 327 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683172 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910694 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124771 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090072 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363497 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116978 s, 89.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (22s) == sanityn test 77f: check TBF JobID nrs policy ========== 21:42:21 (1767321741) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444367 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597988 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717292 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722195 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151371 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196817 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120984 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100397 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195457 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258558 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254529 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253294 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234477 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289911 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247602 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239977 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261528 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01752 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348962 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251178 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247363 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250083 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333179 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341534 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380465 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313031 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451207 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10315 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108757 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610514 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815544 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641829 s, 16.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967963 s, 10.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.332462 s, 50.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592664 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877768 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101373 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189732 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263573 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268114 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257494 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273269 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266622 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342552 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347031 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366875 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341347 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301092 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320089 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384843 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419845 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688992 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923512 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633843 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547565 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453755 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616408 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915097 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011235 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038067 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484184 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464055 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422148 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410668 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364375 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324982 s, 32.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.691433 s, 24.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749447 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862017 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104814 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113774 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168002 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237454 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237324 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290612 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296579 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323344 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332702 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355992 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330059 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 5001+0 records in 1+0 records out [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39105 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405152 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44209 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601389 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605239 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611799 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00941032 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068277 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488886 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685793 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691931 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513507 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735033 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103962 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073139 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735265 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111874 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825386 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972961 s, 10.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409241 s, 41.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112272 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119534 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179352 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238731 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270495 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275638 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319635 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283272 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331568 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29478 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273084 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335415 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277753 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296581 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319666 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314714 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023091 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070228 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335988 s, 312 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634108 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233783 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790619 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463097 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481722 s, 21.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623194 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122159 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534708 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336001 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645905 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830593 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588777 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106271 s, 98.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (22s) == sanityn test 77g: Change TBF type directly ============ 21:42:44 (1767321764) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (10s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 21:42:55 (1767321775) oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg317-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg317-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg317-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg317-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 21:43:06 (1767321786) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 21:43:27 (1767321807) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.558078 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554363 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644486 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649526 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553106 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769297 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193647 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114274 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508282 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742923 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757075 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291292 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283248 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253823 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351284 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368893 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369797 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264914 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246356 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213299 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208645 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214278 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289443 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235056 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235196 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222513 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224063 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236524 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292094 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234379 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237668 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240637 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243761 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=2 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60431 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6083 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (61s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 21:44:28 (1767321868) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31888 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903806 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129049 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119898 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170324 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198742 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181877 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206558 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234674 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859948 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247515 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151683 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295902 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280462 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249097 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320669 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347777 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777497 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629621 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835429 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686728 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504458 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884101 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625809 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680709 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106072 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557071 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676926 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688838 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593801 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700869 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744566 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903177 s, 11.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9672 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9369 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29226 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643934 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104643 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109362 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103589 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150309 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145266 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192124 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237847 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248456 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276721 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110064 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261919 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293614 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314681 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111199 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429908 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466037 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683433 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831378 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362433 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626885 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853773 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951271 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953183 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956313 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731606 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827301 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989727 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103494 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735317 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769483 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856134 s, 12.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9997 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9856 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (148s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:46:56 (1767322016) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2748 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893089 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116186 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145145 s, 7.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143054 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112952 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145164 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151376 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14315 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124346 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127154 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168021 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314614 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316984 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318374 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32247 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319133 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103139 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556696 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655646 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310388 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411982 s, 25.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429141 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777687 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663455 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869216 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157682 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886195 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687715 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675115 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152659 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542854 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00555089 s, 189 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8519 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9372 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35557 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127892 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132847 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157392 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18556 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188482 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241361 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340443 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400956 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4342 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425934 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419709 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453082 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440101 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419291 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430855 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444402 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549103 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653404 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652914 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702035 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396857 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983777 s, 10.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795037 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102918 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825737 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101745 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100561 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893597 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854872 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112797 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130139 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107146 s, 9.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9499 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9483 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (147s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:49:23 (1767322163) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.500875 s, 33.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100989 s, 10.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424383 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738926 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228094 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144256 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12041 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208418 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106745 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842241 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701627 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676543 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148516 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648076 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163924 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259049 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176155 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499911 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081361 s, 12.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484137 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019227 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613996 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369821 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334529 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625057 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531608 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682372 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681225 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479691 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455118 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752895 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580321 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492642 s, 21.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.62807 s, 15.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89769 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.17@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.17@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434579 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645604 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507327 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286552 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893456 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124674 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732373 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197404 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205654 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223155 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217046 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116092 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247957 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192665 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241621 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133078 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161598 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395994 s, 26.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170243 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257939 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271045 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580186 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284492 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356645 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394667 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471646 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043508 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454174 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509802 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847643 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629237 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639565 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644684 s, 16.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.46575 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92203 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.17@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.547456 s, 30.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110144 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679345 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207119 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119566 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10294 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190921 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21491 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355944 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212284 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350023 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260304 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463479 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524725 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.591214 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409797 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341097 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10132 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134144 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583253 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161954 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375184 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860501 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892424 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581401 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602445 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684916 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736223 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888016 s, 11.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912974 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967092 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996165 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114887 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.61615 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14962 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.67711 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261649 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244881 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312643 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101074 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847267 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137672 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126735 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195886 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127067 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202196 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24783 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242679 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307408 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353786 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390876 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387541 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485814 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117276 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574547 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101757 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123689 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379862 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797905 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103811 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722225 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372988 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488881 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031912 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260068 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398647 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232352 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292166 s, 35.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85062 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.03058 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94145 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.914678 s, 18.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880123 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293669 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725291 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113796 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102252 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103734 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137115 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102012 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169084 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220021 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870029 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26701 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17121 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280936 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311261 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307012 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527483 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411601 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478395 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829356 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05324 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106636 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00314421 s, 333 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004486 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465662 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628495 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444498 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558847 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471847 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914813 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071949 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756654 s, 139 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8718 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5656 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402978 s, 41.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572501 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845128 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713335 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146135 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191635 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192852 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202527 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217762 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239395 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206893 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269675 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295066 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229484 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275976 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287861 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296531 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688622 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331554 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00207528 s, 505 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696904 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156334 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247657 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301916 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100208 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459051 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070675 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00212146 s, 494 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00809205 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00273851 s, 383 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117548 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861387 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121216 s, 86.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.33889 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8965 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5749 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458067 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649865 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089666 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170425 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177631 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165376 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141177 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140202 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209638 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19461 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234211 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201259 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194586 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1602 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253398 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272617 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030086 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227606 s, 461 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00216386 s, 485 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251268 s, 417 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443022 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373278 s, 281 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105647 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304758 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931933 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104335 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007966 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622876 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363883 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03633 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233838 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299579 s, 35.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9757 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97314 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2497 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94707 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (411s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:56:14 (1767322574) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.437474 s, 38.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869698 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095533 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957053 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785022 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973758 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166619 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100353 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068228 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193978 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155038 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112924 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150246 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24999 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330806 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335347 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368083 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019748 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379191 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707416 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912646 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605953 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615558 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545751 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704541 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604007 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078306 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737731 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072041 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747894 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798547 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819822 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869398 s, 12.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46353 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89138 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (46s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:57:00 (1767322620) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.17@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.17@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.540453 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907847 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517643 s, 20.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097452 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937549 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223944 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179946 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240906 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207814 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269239 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195502 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195583 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278279 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196963 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248179 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262417 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285644 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897207 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990425 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128916 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108627 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127556 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130009 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16171 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143035 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17031 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117918 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134401 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124162 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154467 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160529 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138261 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189868 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=3 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.52491 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90221 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (45s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:57:45 (1767322665) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.17@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.54047 s, 31.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373075 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524882 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673037 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817485 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104699 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168222 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142672 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226173 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244859 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223776 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186494 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250477 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261703 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238778 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26197 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304902 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d57c2bba800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d57c2bba800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158921 s, 66.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349523 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203383 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218351 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214028 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254454 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264952 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324184 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028723 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664288 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488035 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034153 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359675 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376488 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404295 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467576 s, 22.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.15635 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1135 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (42s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:58:27 (1767322707) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569957 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434709 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676666 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598936 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067923 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109612 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135084 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858256 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132045 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137053 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112151 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165618 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178531 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224761 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225855 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170349 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231308 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634636 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00768901 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611734 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419617 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306229 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510984 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342001 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111649 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426601 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187118 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547374 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577485 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395928 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169611 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263085 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462365 s, 22.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1476 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86971 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.2386 s, 16.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87755 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (85s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:59:53 (1767322793) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.943902 s, 17.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569277 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650379 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829303 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103472 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145994 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880793 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174055 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103809 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451345 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953872 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244495 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197396 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183486 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265545 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1364 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272631 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201294 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604103 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052186 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989807 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823971 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014385 s, 72.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749022 s, 14.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00758068 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674262 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804537 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517251 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483891 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539968 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361242 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980533 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456655 s, 23.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6491 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9941 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (70s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 22:01:02 (1767322862) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436206 s, 38.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452843 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502833 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388045 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914536 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802681 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113832 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955978 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119472 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139523 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171546 s, 6.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177714 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140363 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171328 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144042 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158718 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234039 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029087 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274695 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301197 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318522 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346021 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311943 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390434 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435352 s, 24.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697176 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400306 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465646 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530146 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479307 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525639 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049027 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612112 s, 17.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.01102 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87528 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg317-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503536 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380429 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699015 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882898 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197461 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189361 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206675 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223784 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188567 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249368 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253727 s, 4.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241128 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20156 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264066 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225541 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278395 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228408 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00204426 s, 513 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528319 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00503097 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112113 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488073 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783625 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917975 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587976 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809991 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186885 s, 56.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688947 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839594 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554557 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337155 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114596 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822321 s, 12.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2566 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86128 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0274 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87365 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (126s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 22:03:08 (1767322988) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre opts: Stopping client oleg317-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg317-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.17,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg317-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.17,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg317-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.17,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/58918/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.17,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg317-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0001-super.width=65536 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0001 Started lustre-OST0000 oleg317-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 oleg317-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Started clients oleg317-client.virtnet: 192.168.203.117@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d57f1244800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d57f1245800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57f1244800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57f1245800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.405803 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726526 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379697 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832373 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126425 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134987 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174135 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1862 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227173 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171622 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167386 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244579 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223997 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239908 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218289 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261199 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278166 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603413 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498331 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538397 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721321 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688374 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540079 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751867 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653576 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745166 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760933 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778892 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817449 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900003 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983029 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102163 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11544 s, 9.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.02322 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09399 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.380551 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364336 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540596 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120875 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149514 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133282 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148115 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145489 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15344 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165151 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144609 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164642 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15152 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153497 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14327 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163331 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174555 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555002 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623768 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057261 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053882 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605949 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593564 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550081 s, 19.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462668 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425133 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814308 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638063 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557947 s, 18.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538773 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658239 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688525 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652303 s, 16.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg317-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg317-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.09555 s, 17.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90855 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (179s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 22:06:08 (1767323168) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg317-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (14s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 22:06:22 (1767323182) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (8s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 22:06:30 (1767323190) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg317-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.03251 s, 260 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg317-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0836 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (55s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 22:07:25 (1767323245) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.302021 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586255 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437852 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490238 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168376 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157336 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202004 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194596 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228959 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197128 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22454 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190099 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23109 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238891 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370509 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263903 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04414 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242235 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104095 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251359 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828179 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623502 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520096 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725141 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060073 s, 17.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8d57f1245800.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff8d57f1245800.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429608 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00290077 s, 361 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356781 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634486 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399431 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040924 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342711 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934647 s, 112 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38493 s, 19.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01035 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42688 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371593 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069713 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459602 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648179 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502091 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577918 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819758 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526658 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100374 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122636 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914378 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125953 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168929 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160712 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1833 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043905 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441683 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660997 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590098 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390206 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754193 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351444 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647505 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456528 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663998 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525103 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626476 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704782 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056049 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548696 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650195 s, 16.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg317-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg317-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.86307 s, 17.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06026 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (75s) == sanityn test 77o: Changing rank should not panic ====== 22:08:41 (1767323321) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (9s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 22:08:50 (1767323330) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (74s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 22:10:04 (1767323404) mds.MDS.mdt.nrs_policies=tbf oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg317-client: oleg317-server: ssh exited with exit code 36 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg317-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (22s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 22:10:26 (1767323426) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269708 kill md_thread xargs PASS 77r (43s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 22:11:09 (1767323469) oleg317-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg317-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg317-client: oleg317-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (5s) == sanityn test 79: xattr: intent error ================== 22:11:14 (1767323474) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826934705 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 22:11:15.000000000 -0500 Modify: 2026-01-01 22:11:15.000000000 -0500 Change: 2026-01-01 22:11:16.000000000 -0500 Birth: 2026-01-01 22:11:16.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (15s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 22:11:29 (1767323489) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 22:11:31 (1767323491) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 22:11:33 (1767323493) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 22:11:35 (1767323495) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 22:11:37 (1767323497) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 22:11:39 (1767323499) - open/close 1208 (time 1767323512.33 total 10.00 last 120.78) total: 2500 open/close in 19.86 seconds: 125.91 ops/second snapshot_time 1767323565.367418834 secs.nsecs start_time 1767323500.855832495 secs.nsecs elapsed_time 64.511586339 secs.nsecs open 2505 samples [usecs] 164 9426 2062211 2418621925 close 2504 samples [usecs] 98 8615 726439 474071677 mknod 2501 samples [usecs] 310 9354 1780976 1801744588 unlink 2500 samples [usecs] 341 62783 2863583 16365266373 mkdir 2 samples [usecs] 951 1178 2129 2292085 rmdir 2 samples [usecs] 4133 6215 10348 55707914 rename 2500 samples [usecs] 293 23108 8285907 51273422757 getattr 3764 samples [usecs] 6 1956 122925 25525861 parallel_rename_file 2500 samples [usecs] 293 23108 8285907 51273422757 crossdir_rename 2500 samples [usecs] 293 23108 8285907 51273422757 PASS 81d (69s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 22:12:48 (1767323568) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.9350 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.9350 PASS 82 (4s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 22:12:52 (1767323572) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 22:12:54 (1767323574) fail_loc=0x8000060b PASS 84 (9s) == sanityn test 85: Lustre API root cache race =========== 22:13:03 (1767323583) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1767323584 Finishing test test1 at 1767323585 PASS 85 (5s) == sanityn test 90: open/create and unlink striped directory ========================================================== 22:13:08 (1767323588) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 22:13:09 (1767323589) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 22:13:11 (1767323591) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 22:13:13 (1767323593) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5310 0x14be 0x240000400 0 5346 0x14e2 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5311 0x14bf 0x240000400 0 5347 0x14e3 0x280000400 rc1=0 and rc2=0 PASS 93 (12s) == sanityn test 94: signal vs CP callback race =========== 22:13:24 (1767323604) osc.lustre-OST0000-osc-ffff8d57f1244800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8d57f1245800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d57f1244800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d57f1245800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.196173 s, 41.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.12614 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 284897 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0374485 s, 109 kB/s osc.lustre-OST0000-osc-ffff8d57f1244800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d57f1245800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57f1244800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57f1245800.idle_timeout=debug PASS 94 (13s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 22:13:37 (1767323617) llite.lustre-ffff8d57f1244800.fast_read=0 llite.lustre-ffff8d57f1245800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0405267 s, 51.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8d57f1244800.fast_read=1 llite.lustre-ffff8d57f1245800.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 22:13:53 (1767323633) llite.lustre-ffff8d57f1244800.fast_read=0 llite.lustre-ffff8d57f1245800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00308435 s, 4.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8d57f1244800.fast_read=1 llite.lustre-ffff8d57f1245800.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 22:14:02 (1767323642) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 22:14:04 (1767323644) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0055049 s, 744 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 22:14:07 (1767323647) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100868 s, 20.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (4s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 22:14:11 (1767323651) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0861026 s, 24.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 22:14:15 (1767323655) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00469533 s, 2.6 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 22:14:18 (1767323658) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00332709 s, 1.2 MB/s PASS 101a (4s) == sanityn test 101b: Discard DoM data on rename ========= 22:14:23 (1767323663) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.015401 s, 266 kB/s PASS 101b (4s) == sanityn test 101c: Discard DoM data on close-unlink === 22:14:27 (1767323667) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00662588 s, 618 kB/s PASS 101c (5s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 22:14:33 (1767323673) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1767323675 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1767323678 Finishing test test23 at 1767323678 PASS 103 (9s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 22:14:42 (1767323682) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 22:14:44 (1767323684) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 22:15:08 (1767323708) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 22:15:10 (1767323710) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0134572 s, 76.1 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937240 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 22:15:11.000000000 -0500 Modify: 2026-01-01 22:15:11.000000000 -0500 Change: 2026-01-01 22:15:11.000000000 -0500 Birth: 2026-01-01 22:15:11.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937240 -rw-r--r-- 1 0 0 1767323711 1767323711 1767323711 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937240 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 22:15:11.000000000 -0500 Modify: 2026-01-01 22:15:11.000000000 -0500 Change: 2026-01-01 22:15:11.000000000 -0500 Birth: 2026-01-01 22:15:11.000000000 -0500 1767323711 1024 1 PASS 106b (4s) == sanityn test 106c: Verify statx attributes mask ======= 22:15:14 (1767323714) PASS 106c (4s) == sanityn test 107a: Basic grouplock conflict =========== 22:15:18 (1767323718) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.143676 s, 73.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5355 0x14eb 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 22:15:23 (1767323723) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.156419 s, 67.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5318 0x14c6 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.9350 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 22:15:33 (1767323733) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00360462 s, 2.8 MB/s llite.lustre-ffff8d57f1244800.hybrid_io=0 llite.lustre-ffff8d57f1245800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.39342 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 22:15:40 (1767323740) 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.117@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 PASS 109 (595s) == sanityn test 110: do not grant another lock on resend ========================================================== 22:25:35 (1767324335) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 22:25:36.000000000 -0500 Modify: 2026-01-01 22:25:36.000000000 -0500 Change: 2026-01-01 22:25:36.000000000 -0500 Birth: 2026-01-01 22:25:36.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg317-client.virtnet oleg317-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 22:26:40 (1767324400) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 22:26:42 (1767324402) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 22:26:43 (1767324403) lustre-OST0000-osc-ffff8d57d0733000 active. lustre-OST0001-osc-ffff8d57d0733000 active. lustre-OST0000-osc-ffff8d57d0733000 active. lustre-OST0001-osc-ffff8d57d0733000 active. MGC192.168.203.117@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 22:26:46 (1767324406) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 22:26:48 (1767324408) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 22:26:49 (1767324409) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 22:26:50 (1767324410) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00678815 s, 3.0 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0280259 s, 731 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (6s) == sanityn test 200: service remains healthy while able to process request ========================================================== 22:26:56 (1767324416) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.117 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1431 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.141 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1391 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1587 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.142 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1345 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1756 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2004 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1979 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2479 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2223 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2275 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2461 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2929 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2829 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3047 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3106 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d57c67dd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57c67dd800.ost_server_uuid in IDLE state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 9453 sec ============== 22:28:04 (1767324484) === sanityn: start cleanup 22:28:05 (1767324485) === Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg317-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 22:29:28 (1767324568) ===