-----============= acceptance-small: sanityn ============----- Mon Mar 16 09:38:25 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg152-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 09:38:36 (1773668316) === Starting client oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Mount client oleg152-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Started clients oleg152-client.virtnet: 192.168.201.152@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg152-client.virtnet: executing check_config_client /mnt/lustre oleg152-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg152-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95bd46192800.idle_timeout=debug osc.lustre-OST0000-osc-ffff95bd4a140800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd46192800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd4a140800.idle_timeout=debug disable quota as required oleg152-server: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 09:38:53 (1773668333) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206217 s, 50.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7572] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 09:38:54 (1773668334) PASS 0 (4s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 09:38:59 (1773668339) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 09:39:02 (1773668342) -rw-r--r-- 1 root root 0 Mar 16 09:39 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (3s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 09:39:05 (1773668345) -rw-r--r-- 1 root root 0 Mar 16 09:39 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 09:39:09 (1773668349) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 09:39:12 (1773668352) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 09:39:14 (1773668354) total 1 -rwxrwxrwx 1 root root 0 Mar 16 09:39 f2a -rwxrwxrwx 1 root root 0 Mar 16 09:39 f2b total 1 -rwxrwxrwx 1 root root 0 Mar 16 09:39 f2a -rwxrwxrwx 1 root root 0 Mar 16 09:39 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00659192 s, 77.7 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 09:39:17 (1773668357) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 09:39:19 (1773668359) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103777 s, 20.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 09:39:22 (1773668362) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 09:39:25 (1773668365) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 09:39:29 (1773668369) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 09:39:31 (1773668371) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 09:39:34 (1773668374) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 09:39:37 (1773668377) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (3s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 09:39:40 (1773668380) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 09:39:43 (1773668383) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239926 s, 43.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0226904 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 09:39:47 (1773668387) 1+0 records in 1+0 records out 1 byte copied, 0.00543992 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00376547 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00791438 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00422145 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00741592 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00368396 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00453217 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0032704 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00366595 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00354069 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00268471 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00362548 s, 0.3 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 09:39:50 (1773668390) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0037967 s, 809 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0283154 s, 145 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000183995 s, 16.7 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 09:39:53 (1773668393) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 09:39:56 (1773668396) warning: '-runtime' deprecated, use '-t runtime' instead - link 1083 (time 1773668407.25 total 10.00 last 108.26) - link 2120 (time 1773668417.26 total 20.01 last 103.61) - link 3353 (time 1773668427.26 total 30.02 last 123.26) - link 4477 (time 1773668437.26 total 40.02 last 112.36) - link 5756 (time 1773668447.27 total 50.02 last 127.83) - link 6826 (time 1773668457.27 total 60.03 last 106.97) - link 7839 (time 1773668467.28 total 70.04 last 101.19) - link 9165 (time 1773668477.29 total 80.05 last 132.48) - link 10000 (time 1773668483.96 total 86.71 last 125.25) - link 11164 (time 1773668493.96 total 96.72 last 116.38) total: 11614 link in 100.00 seconds: 116.14 ops/second using seed 2877763766 running for 100 seconds - stat 9596 (time 1773668409 ; total 11 ; last 11) - stat 10000 (time 1773668409 ; total 11 ; last 0) - stat 20000 (time 1773668419 ; total 21 ; last 10) - stat 29534 (time 1773668430 ; total 32 ; last 11) - stat 30000 (time 1773668430 ; total 32 ; last 0) - stat 39815 (time 1773668441 ; total 43 ; last 11) - stat 40000 (time 1773668441 ; total 43 ; last 0) - stat 48149 (time 1773668452 ; total 54 ; last 11) - stat 50000 (time 1773668456 ; total 58 ; last 4) - stat 54242 (time 1773668467 ; total 69 ; last 11) - stat 58493 (time 1773668478 ; total 80 ; last 11) - stat 60000 (time 1773668481 ; total 83 ; last 3) - stat 63819 (time 1773668492 ; total 94 ; last 11) total: 66541 stats in 100 seconds: 665.409973 stats/second - unlinked 0 (time 1773668448 ; total 0 ; last 0) - unlinked 10000 (time 1773668538 ; total 90 ; last 90) unlink(/mnt/lustre2/lockdir/lockfile11614) error: No such file or directory total: 11613 unlinks in 113 seconds: 102.769913 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21348) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21350) - No such process PASS 12 (171s) == sanityn test 13: test directory page revocation ======= 09:42:47 (1773668567) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:42:57 (1773668577) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22540 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:43:05 (1773668585) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23120 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:43:12 (1773668592) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23698 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:43:20 (1773668600) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24280 Terminated $DIR1/$tdir/sleep 60 PASS 14c (12s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:43:32 (1773668612) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24863 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:43:41 (1773668621) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (44s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:44:25 (1773668665) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.313688 s, 26.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.271231 s, 30.9 MB/s Chance of close/open is 1 in 50 Seed set to 8668 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x856248 000001[0] 1773668668.708549 trunc from 00000000 to 0x856247 (0x856248 bytes) 000003[0] 1773668668.737954 write 0x0be76d thru 0x0c7da3 (0x9637 bytes) 000004[1] 1773668668.772978 trunc from 0x856248 to 0x299ddc (0x5bc46b bytes) 000006[0] 1773668668.862607 write 0x2928d2 thru 0x29b27d (0x89ac bytes) 000007[0] 1773668668.893658 trunc from 0x29b27e to 0x68a18f (0x3eef12 bytes) 000008[1] 1773668668.923949 write 0x179aea thru 0x181f4c (0x8463 bytes) 000010[0] 1773668668.983189 write 0x821641 thru 0x82a34f (0x8d0f bytes) 000011[0] 1773668669.008526 mapread 0x19b26f thru 0x1a8c97 (0xda29 bytes) 000018[1] 1773668669.128320 mapwrite 0x8a48fe thru 0x8b0165 (0xb868 bytes) 000024[1] 1773668669.385582 mapread 0x48a86a thru 0x495717 (0xaeae bytes) 000025[1] 1773668669.417160 read 0x889d4b thru 0x892521 (0x87d7 bytes) 000026[1] 1773668669.425848 read 0x511170 thru 0x51e811 (0xd6a2 bytes) 000027[1] 1773668669.433214 trunc from 0x8b0166 to 0x4e392b (0x3cc83a bytes) 000028[1] 1773668669.467578 mapread 0x3c29a8 thru 0x3cebd9 (0xc232 bytes) 000029[1] 1773668669.523299 mapread 0x055689 thru 0x058056 (0x29ce bytes) 000031[0] 1773668669.548737 trunc from 0x4e392c to 0x58ccff (0xa93d4 bytes) 000032[1] 1773668669.608607 mapread 0x082ff9 thru 0x090a43 (0xda4b bytes) 000033[1] 1773668669.684542 mapread 0x161869 thru 0x16d2c5 (0xba5d bytes) 000041[1] 1773668669.824037 trunc from 0x58cd00 to 0x3e0156 (0x1acba9 bytes) 000042[0] 1773668669.875231 write 0x9c0054 thru 0x9c3eb8 (0x3e65 bytes) 000043[0] 1773668669.914108 write 0x253edd thru 0x25e5d1 (0xa6f5 bytes) 000045[0] 1773668669.955195 write 0x3e9974 thru 0x3ecd50 (0x33dd bytes) 000046[1] 1773668669.976482 mapread 0x5293ed thru 0x532101 (0x8d15 bytes) 000051[0] 1773668670.209972 write 0x24a0ea thru 0x253e4d (0x9d64 bytes) 000055[0] 1773668670.255712 mapwrite 0x41febf thru 0x42865c (0x879e bytes) 000056[1] 1773668670.408544 read 0x3d1dde thru 0x3db867 (0x9a8a bytes) 000057[0] 1773668670.470121 mapwrite 0x11b3d3 thru 0x12792c (0xc55a bytes) 000058[1] 1773668670.586990 mapread 0x00adb7 thru 0x01a4b9 (0xf703 bytes) 000061[0] 1773668670.676528 mapwrite 0x662ca1 thru 0x668fe2 (0x6342 bytes) 000062[0] 1773668670.826717 trunc from 0x9c3eb9 to 0x696f40 (0x32cf78 bytes) 000065[1] 1773668670.896155 mapread 0x2d1b57 thru 0x2de545 (0xc9ef bytes) 000068[0] 1773668670.947793 mapwrite 0x879fa7 thru 0x87e5f8 (0x4652 bytes) 000069[1] 1773668671.060557 trunc from 0x87e5f9 to 0x2746d2 (0x609f26 bytes) 000070[0] 1773668671.098596 mapread 0x063dc2 thru 0x06d08c (0x92cb bytes) 000071[0] 1773668671.166536 mapwrite 0x8366c7 thru 0x83b044 (0x497e bytes) 000072[0] 1773668671.308835 mapwrite 0x5420e1 thru 0x54dcfb (0xbc1b bytes) 000073[0] 1773668671.429825 read 0x3c44f8 thru 0x3d3ed6 (0xf9df bytes) 000074[0] 1773668671.454744 trunc from 0x83b045 to 0x73e2eb (0xfcd59 bytes) 000078[1] 1773668671.525869 mapread 0x2b9110 thru 0x2c2b71 (0x9a62 bytes) 000079[0] 1773668671.549763 write 0x19cfbc thru 0x1ac6a1 (0xf6e6 bytes) 000081[0] 1773668671.581677 write 0x84c69a thru 0x851c00 (0x5567 bytes) 000082[1] 1773668671.598996 trunc from 0x851c01 to 0x81396d (0x3e293 bytes) 000084[0] 1773668671.667436 write 0x44008a thru 0x4413f4 (0x136b bytes) 000085[0] 1773668671.697012 read 0x6fc133 thru 0x6fd59d (0x146b bytes) 000086[1] 1773668671.721778 read 0x7a328e thru 0x7ad14f (0x9ec2 bytes) 000088[0] 1773668671.752552 read 0x079d62 thru 0x088a19 (0xecb8 bytes) 000089[1] 1773668671.779200 write 0x0aa20e thru 0x0b7391 (0xd184 bytes) 000090[1] 1773668671.791503 mapread 0x42ce38 thru 0x437796 (0xa95f bytes) 000091[0] 1773668671.819168 mapwrite 0x2c4b35 thru 0x2d3580 (0xea4c bytes) 000092[0] 1773668671.929707 mapwrite 0x65c56b thru 0x666d55 (0xa7eb bytes) 000093[1] 1773668672.000437 write 0x57279c thru 0x57ec2d (0xc492 bytes) 000094[1] 1773668672.047587 read 0x089ffe thru 0x092cdc (0x8cdf bytes) 000095[0] 1773668672.065904 trunc from 0x81396e to 0x168b2d (0x6aae40 bytes) 000096[1] 1773668672.135876 mapread 0x0f6a45 thru 0x100d7a (0xa336 bytes) truncating to largest ever: 0x8fa374 000097[1] 1773668672.182346 trunc from 0x168b2e to 0x8fa373 (0x791846 bytes) 000100[1] 1773668672.247390 write 0x5537d2 thru 0x55b068 (0x7897 bytes) 000103[1] 1773668672.270804 write 0x2d38b1 thru 0x2d4c73 (0x13c3 bytes) 000104[1] 1773668672.278944 read 0x501a56 thru 0x506ca0 (0x524b bytes) 000107[1] 1773668672.300437 trunc from 0x8fa374 to 0x59d30a (0x35d069 bytes) 000110[0] 1773668672.336331 mapread 0x03a38b thru 0x0417d3 (0x7449 bytes) 000111[1] 1773668672.399584 mapread 0x1c47d8 thru 0x1d0ce0 (0xc509 bytes) 000112[1] 1773668672.458722 mapwrite 0x02e1c0 thru 0x03cb32 (0xe973 bytes) 000113[0] 1773668672.582337 mapwrite 0x211764 thru 0x21783a (0x60d7 bytes) 000114[0] 1773668672.720607 trunc from 0x59d30b to 0x644385 (0xa707b bytes) 000115[0] 1773668672.776475 read 0x381bce thru 0x387417 (0x584a bytes) 000118[0] 1773668672.821785 write 0x8170b1 thru 0x82644e (0xf39e bytes) 000119[0] 1773668672.825540 mapread 0x630e3e thru 0x6329a2 (0x1b65 bytes) 000121[1] 1773668672.893328 mapread 0x7223e6 thru 0x72e3c0 (0xbfdb bytes) 000122[0] 1773668673.138649 mapread 0x2a30d4 thru 0x2a552c (0x2459 bytes) 000123[0] 1773668673.188358 read 0x287fc4 thru 0x2880d0 (0x10d bytes) 000124[1] 1773668673.195492 mapread 0x769048 thru 0x769b7e (0xb37 bytes) 000131[1] 1773668673.295850 mapwrite 0x6f85c8 thru 0x6ff0d6 (0x6b0f bytes) 000132[0] 1773668673.682089 mapwrite 0x414edc thru 0x41eaf3 (0x9c18 bytes) 000136[1] 1773668673.847409 mapread 0x22a59b thru 0x22f871 (0x52d7 bytes) 000137[1] 1773668673.934322 trunc from 0x82644f to 0x58d2a9 (0x2991a5 bytes) 000140[0] 1773668674.025092 mapread 0x443f88 thru 0x45136a (0xd3e3 bytes) 000141[1] 1773668674.075307 read 0x458831 thru 0x46830b (0xfadb bytes) 000146[0] 1773668674.111017 trunc from 0x58d2aa to 0x4dfb12 (0xad797 bytes) 000149[0] 1773668674.227919 write 0x951ef7 thru 0x953671 (0x177b bytes) 000151[1] 1773668674.237662 read 0x192912 thru 0x1998b8 (0x6fa7 bytes) 000153[1] 1773668674.295360 mapread 0x8c5327 thru 0x8cbbde (0x68b8 bytes) 000154[0] 1773668674.329973 read 0x6fefc6 thru 0x70cb2b (0xdb66 bytes) 000156[1] 1773668674.444665 mapread 0x7b59cf thru 0x7badb3 (0x53e5 bytes) 000158[1] 1773668674.511189 trunc from 0x953672 to 0x245649 (0x70e028 bytes) 000159[0] 1773668674.553343 write 0x287458 thru 0x28a9b0 (0x3559 bytes) 000161[0] 1773668674.603471 mapread 0x02c5e3 thru 0x031a9e (0x54bc bytes) 000163[1] 1773668674.689661 mapread 0x1e34ce thru 0x1e992a (0x645d bytes) 000164[1] 1773668674.722297 mapread 0x208026 thru 0x20ca42 (0x4a1d bytes) 000165[0] 1773668674.757453 write 0x595aa7 thru 0x5a27d0 (0xcd2a bytes) 000166[1] 1773668674.778453 mapwrite 0x8a011f thru 0x8af965 (0xf847 bytes) 000168[0] 1773668674.941367 write 0x712c83 thru 0x715ba6 (0x2f24 bytes) 000171[0] 1773668675.004215 write 0x239df5 thru 0x23a7aa (0x9b6 bytes) 000172[0] 1773668675.076868 trunc from 0x8af966 to 0x3366ff (0x579266 bytes) 000173[1] 1773668675.139226 read 0x28a0ba thru 0x28bcb4 (0x1bfb bytes) 000176[0] 1773668675.282910 write 0x780618 thru 0x78fa02 (0xf3eb bytes) 000177[0] 1773668675.309575 mapread 0x0c81c0 thru 0x0cd30e (0x514f bytes) 000180[0] 1773668675.379512 mapread 0x665e97 thru 0x66c7b8 (0x6922 bytes) 000182[1] 1773668675.419447 read 0x080abf thru 0x081537 (0xa79 bytes) 000185[1] 1773668675.466718 mapwrite 0x85ed45 thru 0x85f9ea (0xca6 bytes) truncating to largest ever: 0x913ff5 000186[0] 1773668675.614734 trunc from 0x85f9eb to 0x913ff4 (0xb460a bytes) 000187[1] 1773668675.644009 trunc from 0x913ff5 to 0x8be615 (0x559df bytes) 000188[0] 1773668675.682846 trunc from 0x8be616 to 0x656393 (0x268282 bytes) 000189[0] 1773668675.726626 write 0x9b9633 thru 0x9c2854 (0x9222 bytes) 000190[0] 1773668675.764249 read 0x3f2da4 thru 0x3fc699 (0x98f6 bytes) 000191[0] 1773668675.809650 mapread 0x5f315d thru 0x601e6e (0xed12 bytes) 000193[0] 1773668675.901673 mapread 0x264f08 thru 0x273b4f (0xec48 bytes) 000194[0] 1773668675.970454 trunc from 0x9c2855 to 0x79c9e1 (0x225e73 bytes) 000195[1] 1773668676.015958 write 0x799d16 thru 0x79b6a5 (0x1990 bytes) 000196[1] 1773668676.070660 read 0x5b59f3 thru 0x5be13f (0x874d bytes) 000197[1] 1773668676.109341 write 0x7d3f52 thru 0x7da4ed (0x659c bytes) 000198[0] 1773668676.126320 trunc from 0x7da4ee to 0x223239 (0x5b72b4 bytes) 000201[0] 1773668676.247233 mapread 0x0514de thru 0x05225f (0xd82 bytes) 000203[1] 1773668676.274464 mapread 0x1137f6 thru 0x121d70 (0xe57b bytes) 000204[1] 1773668676.353269 write 0x4a9d53 thru 0x4b828c (0xe53a bytes) 000205[0] 1773668676.368347 read 0x3f8ea6 thru 0x3fbfb5 (0x3110 bytes) 000206[0] 1773668676.424848 write 0x180342 thru 0x18ead3 (0xe792 bytes) 000207[0] 1773668676.467349 mapwrite 0x7f2d31 thru 0x8020ab (0xf37b bytes) truncating to largest ever: 0x9e1b59 000208[1] 1773668676.702701 trunc from 0x8020ac to 0x9e1b58 (0x1dfaad bytes) 000209[0] 1773668676.738352 read 0x39299e thru 0x393470 (0xad3 bytes) 000210[1] 1773668676.780237 trunc from 0x9e1b59 to 0x67dbd4 (0x363f84 bytes) 000211[0] 1773668676.851734 mapwrite 0x301e09 thru 0x30a772 (0x896a bytes) 000215[1] 1773668677.025222 read 0x2729b6 thru 0x27b787 (0x8dd2 bytes) 000216[1] 1773668677.046922 write 0x677bb7 thru 0x68384e (0xbc98 bytes) 000221[0] 1773668677.127465 read 0x65e285 thru 0x66d250 (0xefcc bytes) 000223[1] 1773668677.177080 write 0x61f94c thru 0x623348 (0x39fd bytes) 000226[0] 1773668677.228792 mapread 0x4b8de4 thru 0x4ba4e4 (0x1701 bytes) 000227[0] 1773668677.281839 write 0x90ce42 thru 0x91035d (0x351c bytes) 000229[0] 1773668677.326048 read 0x1763d6 thru 0x17af5b (0x4b86 bytes) 000232[0] 1773668677.368843 read 0x211045 thru 0x21af2d (0x9ee9 bytes) 000233[0] 1773668677.376962 trunc from 0x91035e to 0x8ff3a9 (0x10fb4 bytes) 000234[0] 1773668677.455110 write 0x2181a8 thru 0x226561 (0xe3ba bytes) 000235[0] 1773668677.466219 mapread 0x713270 thru 0x71b6c6 (0x8457 bytes) 000236[0] 1773668677.537647 mapwrite 0x841c6a thru 0x84b33e (0x96d5 bytes) 000241[0] 1773668677.696165 trunc from 0x8ff3aa to 0x408852 (0x4f6b57 bytes) 000243[0] 1773668677.750773 write 0x10f818 thru 0x115b48 (0x6331 bytes) 000249[1] 1773668677.831148 trunc from 0x408853 to 0x4c259e (0xb9d4c bytes) 000250[1] 1773668677.901435 write 0x43cb25 thru 0x44844c (0xb928 bytes) 000255[0] 1773668677.963661 mapread 0x044e8d thru 0x04a245 (0x53b9 bytes) 000256[0] 1773668677.995161 mapwrite 0x2edabb thru 0x2fa8ec (0xce32 bytes) 000258[0] 1773668678.151064 trunc from 0x4c259f to 0x002185 (0x4c0419 bytes) 000259[0] 1773668678.216252 mapread 0x00016a thru 0x002185 (0x201c bytes) 000260[0] 1773668678.261443 mapwrite 0x492a49 thru 0x499512 (0x6aca bytes) 000262[0] 1773668678.394350 read 0x08d5f0 thru 0x09007f (0x2a90 bytes) 000263[0] 1773668678.421604 trunc from 0x499513 to 0x8e2b96 (0x449684 bytes) 000265[1] 1773668678.502607 mapwrite 0x7749ff thru 0x780e17 (0xc419 bytes) 000268[1] 1773668678.692776 mapwrite 0x36fe76 thru 0x378e76 (0x9001 bytes) 000269[1] 1773668678.758537 trunc from 0x8e2b97 to 0x4aec6e (0x433f28 bytes) 000270[1] 1773668678.777559 trunc from 0x4aec6f to 0x7cc954 (0x31dce6 bytes) 000275[1] 1773668678.824054 write 0x357a3c thru 0x3672b7 (0xf87c bytes) 000276[1] 1773668678.846906 mapread 0x260a58 thru 0x26d5a2 (0xcb4b bytes) 000278[1] 1773668678.885349 write 0x8b27f5 thru 0x8b95ca (0x6dd6 bytes) 000282[1] 1773668678.921228 mapread 0x6a937a thru 0x6acd45 (0x39cc bytes) 000283[1] 1773668678.976923 trunc from 0x8b95cb to 0x03d887 (0x87bd43 bytes) 000285[0] 1773668678.998065 mapread 0x03357b thru 0x03d887 (0xa30d bytes) 000286[0] 1773668679.049840 read 0x029a03 thru 0x031676 (0x7c74 bytes) 000288[1] 1773668679.062923 mapread 0x012725 thru 0x01eeaf (0xc78b bytes) 000289[1] 1773668679.087944 read 0x017552 thru 0x01b0e3 (0x3b92 bytes) 000290[0] 1773668679.104529 trunc from 0x03d888 to 0x75549e (0x717c17 bytes) 000291[1] 1773668679.156590 write 0x31df0c thru 0x326b70 (0x8c65 bytes) 000293[1] 1773668679.261580 mapwrite 0x7be0ef thru 0x7cd964 (0xf876 bytes) 000295[0] 1773668679.518721 trunc from 0x7cd965 to 0x4a5a33 (0x327f31 bytes) 000296[1] 1773668679.591741 mapread 0x102dac thru 0x10dfa2 (0xb1f7 bytes) 000297[1] 1773668679.692068 mapread 0x0df7d2 thru 0x0e2f7a (0x37a9 bytes) 000298[1] 1773668679.740146 trunc from 0x4a5a34 to 0x8b6835 (0x410e02 bytes) 000299[1] 1773668679.800137 write 0x35ffde thru 0x36c4d3 (0xc4f6 bytes) 000303[0] 1773668679.847536 write 0x1acaf8 thru 0x1bbf62 (0xf46b bytes) 000304[1] 1773668679.938691 trunc from 0x8b6836 to 0x32494b (0x591eea bytes) 000310[1] 1773668680.097664 mapread 0x318f0f thru 0x32494b (0xba3d bytes) 000313[0] 1773668680.199053 read 0x02697b thru 0x032be4 (0xc26a bytes) 000314[0] 1773668680.324036 mapwrite 0x221ebd thru 0x22c100 (0xa244 bytes) 000315[0] 1773668680.508704 trunc from 0x32494c to 0x004450 (0x3204fb bytes) 000319[1] 1773668680.624316 write 0x3c27f7 thru 0x3ceff1 (0xc7fb bytes) 000323[0] 1773668680.721263 write 0x28f878 thru 0x295886 (0x600f bytes) 000324[0] 1773668680.775159 trunc from 0x3ceff2 to 0x8f1c4c (0x522c5b bytes) 000325[0] 1773668680.834801 write 0x581387 thru 0x5876f0 (0x636a bytes) 000327[1] 1773668680.908600 mapwrite 0x56bd7b thru 0x5749e6 (0x8c6c bytes) 000328[0] 1773668681.061314 mapread 0x55d911 thru 0x56a7b0 (0xcea0 bytes) 000331[1] 1773668681.166264 read 0x49fa75 thru 0x4a05f1 (0xb7d bytes) 000332[1] 1773668681.175262 write 0x6737b9 thru 0x6829ca (0xf212 bytes) 000340[1] 1773668681.380035 write 0x546543 thru 0x54ac7b (0x4739 bytes) 000341[1] 1773668681.419553 read 0x44d180 thru 0x44fa75 (0x28f6 bytes) 000342[0] 1773668681.464455 mapwrite 0x678155 thru 0x687747 (0xf5f3 bytes) 000346[0] 1773668681.708117 write 0x7d6dc2 thru 0x7e2383 (0xb5c2 bytes) 000347[0] 1773668681.750670 read 0x5a503f thru 0x5b1012 (0xbfd4 bytes) 000348[0] 1773668681.758197 trunc from 0x8f1c4d to 0x2dcae7 (0x615165 bytes) 000349[0] 1773668681.809541 write 0x422868 thru 0x426c3d (0x43d6 bytes) 000352[0] 1773668681.854743 write 0x2c8142 thru 0x2cbc6b (0x3b2a bytes) 000357[1] 1773668681.913119 read 0x1c75dc thru 0x1d5208 (0xdc2d bytes) 000361[0] 1773668682.019002 trunc from 0x426c3e to 0x09280b (0x394432 bytes) 000368[1] 1773668682.174619 write 0x63a0cf thru 0x645085 (0xafb7 bytes) 000369[1] 1773668682.226923 mapwrite 0x8ccf7a thru 0x8d8b3a (0xbbc1 bytes) 000370[0] 1773668682.404136 write 0x011cdb thru 0x01ff11 (0xe237 bytes) 000372[1] 1773668682.459539 read 0x233628 thru 0x23dcb4 (0xa68d bytes) 000375[0] 1773668682.553589 mapwrite 0x216f04 thru 0x21f9c2 (0x8abf bytes) 000376[1] 1773668682.691650 mapwrite 0x166455 thru 0x175ac5 (0xf671 bytes) 000377[0] 1773668682.792718 mapwrite 0x334227 thru 0x334aad (0x887 bytes) 000378[0] 1773668682.908401 write 0x8dcc4e thru 0x8ea34a (0xd6fd bytes) 000380[1] 1773668682.967925 mapread 0x57d350 thru 0x583b46 (0x67f7 bytes) 000383[0] 1773668683.076690 mapread 0x24dc71 thru 0x24ddc6 (0x156 bytes) 000384[1] 1773668683.125404 write 0x68489e thru 0x684e93 (0x5f6 bytes) 000387[0] 1773668683.210260 mapwrite 0x528ba5 thru 0x52edef (0x624b bytes) 000388[0] 1773668683.380300 read 0x507b35 thru 0x50f7d8 (0x7ca4 bytes) 000389[0] 1773668683.381895 read 0x290049 thru 0x298284 (0x823c bytes) 000391[0] 1773668683.413802 read 0x862233 thru 0x86829a (0x6068 bytes) 000392[1] 1773668683.439909 mapwrite 0x6695f9 thru 0x67017b (0x6b83 bytes) 000396[0] 1773668683.631324 mapwrite 0x57a18b thru 0x57c3fc (0x2272 bytes) 000397[1] 1773668683.757836 read 0x6bc9e5 thru 0x6c59d5 (0x8ff1 bytes) 000400[0] 1773668683.812218 trunc from 0x8ea34b to 0x90b6ba (0x21370 bytes) 000407[0] 1773668684.033612 mapread 0x21689a thru 0x224032 (0xd799 bytes) 000410[1] 1773668684.070967 mapwrite 0x6d2124 thru 0x6dca53 (0xa930 bytes) 000411[1] 1773668684.172302 write 0x268a7e thru 0x26ff15 (0x7498 bytes) 000413[0] 1773668684.237235 trunc from 0x90b6bb to 0x1b1803 (0x759eb7 bytes) 000416[0] 1773668684.341603 mapread 0x042bfb thru 0x045d4e (0x3154 bytes) 000417[0] 1773668684.392997 write 0x3d5e3b thru 0x3de03e (0x8204 bytes) 000420[0] 1773668684.461470 write 0x2050a6 thru 0x20ab7e (0x5ad9 bytes) 000422[0] 1773668684.494692 write 0x864ed1 thru 0x8703bd (0xb4ed bytes) 000424[1] 1773668684.523661 mapwrite 0x6909cd thru 0x694d33 (0x4367 bytes) 000425[1] 1773668684.841983 write 0x903b92 thru 0x911022 (0xd491 bytes) 000427[1] 1773668684.934079 write 0x5ea668 thru 0x5f3a5b (0x93f4 bytes) 000430[0] 1773668685.040087 trunc from 0x911023 to 0x55bfab (0x3b5077 bytes) 000432[1] 1773668685.123901 mapread 0x03d27c thru 0x0492bf (0xc044 bytes) 000436[1] 1773668685.270457 read 0x0586ef thru 0x061161 (0x8a73 bytes) 000438[1] 1773668685.271828 mapread 0x03e193 thru 0x047df1 (0x9c5f bytes) 000440[1] 1773668685.280740 mapread 0x3f1719 thru 0x3f68bc (0x51a4 bytes) 000441[0] 1773668685.310530 write 0x13c61a thru 0x141c53 (0x563a bytes) 000443[0] 1773668685.394699 mapwrite 0x05f2d7 thru 0x06c933 (0xd65d bytes) 000444[0] 1773668685.500339 trunc from 0x55bfac to 0x17c20a (0x3dfda1 bytes) 000445[1] 1773668685.532484 read 0x0e6c51 thru 0x0ec7f4 (0x5ba4 bytes) 000447[0] 1773668685.606243 mapread 0x0b54f2 thru 0x0b7e76 (0x2985 bytes) 000449[1] 1773668685.644046 read 0x109efd thru 0x10de9a (0x3f9e bytes) 000451[1] 1773668685.753519 mapwrite 0x258beb thru 0x25ef63 (0x6379 bytes) 000452[0] 1773668685.897355 trunc from 0x25ef64 to 0x78c4f6 (0x52d593 bytes) 000453[0] 1773668685.953912 mapread 0x1a1ab5 thru 0x1af2ca (0xd816 bytes) 000454[1] 1773668685.983245 read 0x664f3b thru 0x67196f (0xca35 bytes) 000455[1] 1773668686.134338 read 0x6656fc thru 0x672dcd (0xd6d2 bytes) 000456[0] 1773668686.148872 mapread 0x2bdbf1 thru 0x2c3d44 (0x6154 bytes) 000457[1] 1773668686.188525 read 0x099abe thru 0x09f657 (0x5b9a bytes) 000460[0] 1773668686.222348 read 0x757f39 thru 0x7647a9 (0xc871 bytes) 000461[0] 1773668686.236572 write 0x2a4fb6 thru 0x2af625 (0xa670 bytes) 000462[1] 1773668686.268880 mapread 0x3ce9b7 thru 0x3dd2ea (0xe934 bytes) 000466[0] 1773668686.341034 mapread 0x1c701a thru 0x1d2a63 (0xba4a bytes) 000467[1] 1773668686.362497 trunc from 0x78c4f7 to 0x7ec81a (0x60324 bytes) 000468[1] 1773668686.451398 write 0x8048a5 thru 0x80779c (0x2ef8 bytes) 000470[1] 1773668686.517070 trunc from 0x80779d to 0x2c18d4 (0x545ec8 bytes) 000472[0] 1773668686.570811 trunc from 0x2c18d5 to 0x0a35e9 (0x21e2eb bytes) 000473[0] 1773668686.620979 mapwrite 0x42c411 thru 0x43220f (0x5dff bytes) 000475[0] 1773668686.730471 trunc from 0x432210 to 0x9c61fe (0x593fef bytes) 000477[1] 1773668686.835963 read 0x8e4b96 thru 0x8eaedb (0x6346 bytes) 000478[0] 1773668686.892152 write 0x75c59b thru 0x764dad (0x8813 bytes) 000480[1] 1773668686.907041 write 0x2b2798 thru 0x2bba22 (0x928b bytes) 000482[1] 1773668686.963322 write 0x513307 thru 0x51cddc (0x9ad6 bytes) 000485[0] 1773668687.039076 read 0x90f4a5 thru 0x911b3e (0x269a bytes) 000488[1] 1773668687.078863 read 0x3770eb thru 0x384e78 (0xdd8e bytes) 000489[1] 1773668687.093254 mapread 0x8388c4 thru 0x839216 (0x953 bytes) 000490[1] 1773668687.144872 trunc from 0x9c61ff to 0x5066fe (0x4bfb00 bytes) 000493[0] 1773668687.226286 read 0x10eacb thru 0x11bfa8 (0xd4de bytes) 000497[1] 1773668687.272481 read 0x4af612 thru 0x4b45be (0x4fad bytes) 000498[0] 1773668687.294316 write 0x3e59dd thru 0x3eb7e9 (0x5e0d bytes) 000500[0] 1773668687.340844 mapread 0x460a76 thru 0x4674fb (0x6a86 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8687 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x69c304 000002[0] 1773668688.000479 trunc from 00000000 to 0x69c303 (0x69c304 bytes) 000003[0] 1773668688.058911 write_OD 0x57e000 thru 0x584fff (0x7000 bytes) 000005[1] 1773668688.119629 mapwrite 0x297000 thru 0x2a11be (0xa1bf bytes) 000006[0] 1773668688.217363 read_OD 0x039000 thru 0x040fff (0x8000 bytes) 000007[0] 1773668688.256037 read_OD 0x65e000 thru 0x66cfff (0xf000 bytes) 000008[1] 1773668688.274481 mapwrite 0x30f000 thru 0x311cd6 (0x2cd7 bytes) 000012[0] 1773668688.403022 mapwrite 0x553000 thru 0x554215 (0x1216 bytes) 000014[1] 1773668688.524070 read 0x58d000 thru 0x591662 (0x4663 bytes) 000015[1] 1773668688.552993 mapwrite 0x2e1000 thru 0x2eda6b (0xca6c bytes) 000017[1] 1773668688.694516 mapwrite 0x961000 thru 0x964315 (0x3316 bytes) 000018[0] 1773668688.804518 write_OD 0x6bb000 thru 0x6c5fff (0xb000 bytes) 000019[0] 1773668688.875266 trunc from 0x964316 to 0x1eaa52 (0x7798c3 bytes) 000020[1] 1773668688.900826 write 0x585000 thru 0x58b98a (0x698b bytes) 000021[1] 1773668688.936810 mapread 0x4dd000 thru 0x4e9522 (0xc523 bytes) 000023[1] 1773668688.988261 mapwrite 0x8f4000 thru 0x8fd410 (0x9411 bytes) 000024[1] 1773668689.101543 read 0x6f1000 thru 0x6f6784 (0x5785 bytes) 000026[1] 1773668689.111174 mapwrite 0x7bd000 thru 0x7c7318 (0xa319 bytes) 000029[0] 1773668689.200126 mapread 0x0d8000 thru 0x0d88bf (0x8c0 bytes) 000031[1] 1773668689.228426 read 0x341000 thru 0x3474c4 (0x64c5 bytes) 000033[1] 1773668689.241590 mapwrite 0x17b000 thru 0x188bb4 (0xdbb5 bytes) 000034[1] 1773668689.363266 mapwrite 0x071000 thru 0x077765 (0x6766 bytes) 000035[1] 1773668689.479008 mapwrite 0x134000 thru 0x142888 (0xe889 bytes) 000042[1] 1773668689.629887 read 0x674000 thru 0x683b4d (0xfb4e bytes) 000044[0] 1773668689.681556 mapread 0x284000 thru 0x289365 (0x5366 bytes) 000045[1] 1773668689.719099 read 0x5e7000 thru 0x5ee6d4 (0x76d5 bytes) 000049[0] 1773668689.727966 mapread 0x8c3000 thru 0x8d158b (0xe58c bytes) 000051[1] 1773668689.770824 mapread 0x08b000 thru 0x094fd6 (0x9fd7 bytes) truncating to largest ever: 0x8691a5 000052[1] 1773668689.808597 trunc from 0x8fd411 to 0x8691a4 (0x9426c bytes) 000053[1] 1773668689.871437 read 0x5fe000 thru 0x601789 (0x378a bytes) 000055[0] 1773668689.922811 write_OD 0x03e000 thru 0x03efff (0x1000 bytes) 000059[1] 1773668690.001293 trunc from 0x8691a5 to 0x1a7bdf (0x6c15c5 bytes) 000060[1] 1773668690.052242 write 0x651000 thru 0x657b7c (0x6b7d bytes) 000066[0] 1773668690.119326 write_OD 0x899000 thru 0x8a7fff (0xf000 bytes) 000067[0] 1773668690.178179 write_OD 0x82e000 thru 0x831fff (0x4000 bytes) 000068[0] 1773668690.221977 write_OD 0x6b5000 thru 0x6c0fff (0xc000 bytes) 000073[0] 1773668690.303264 trunc from 0x8a8000 to 0x7fead3 (0xa952c bytes) 000074[1] 1773668690.354033 write 0x026000 thru 0x02c16e (0x616f bytes) 000075[0] 1773668690.366574 trunc from 0x7fead4 to 0x42ce38 (0x3d1c9b bytes) 000076[1] 1773668690.416199 mapwrite 0x1ab000 thru 0x1b7b19 (0xcb1a bytes) 000078[1] 1773668690.624634 mapread 0x2fa000 thru 0x2fc6ad (0x26ae bytes) 000079[0] 1773668690.659330 read_OD 0x032000 thru 0x03dfff (0xc000 bytes) 000080[1] 1773668690.687534 read 0x081000 thru 0x08977b (0x877c bytes) 000083[0] 1773668690.697434 write_OD 0x877000 thru 0x885fff (0xf000 bytes) 000084[1] 1773668690.752153 trunc from 0x886000 to 0x3b52d5 (0x4d0d2a bytes) 000086[0] 1773668690.839977 write_OD 0x67f000 thru 0x685fff (0x7000 bytes) 000091[0] 1773668690.929663 write_OD 0x8d6000 thru 0x8e0fff (0xb000 bytes) 000092[0] 1773668690.986950 trunc from 0x8e1000 to 0x514ef5 (0x3cc10a bytes) 000094[0] 1773668691.056203 read_OD 0x370000 thru 0x37cfff (0xd000 bytes) 000095[1] 1773668691.080693 write 0x73f000 thru 0x741b93 (0x2b94 bytes) truncating to largest ever: 0x9c3574 000096[0] 1773668691.096612 trunc from 0x741b94 to 0x9c3573 (0x2819e0 bytes) 000098[1] 1773668691.153761 trunc from 0x9c3574 to 0x670f19 (0x35265a bytes) 000100[1] 1773668691.242002 write 0x86e000 thru 0x871039 (0x303a bytes) 000102[0] 1773668691.306536 mapread 0x19a000 thru 0x1a3ee4 (0x9ee5 bytes) 000103[0] 1773668691.393174 mapread 0x215000 thru 0x22247c (0xd47d bytes) 000107[1] 1773668691.434638 mapwrite 0x722000 thru 0x723168 (0x1169 bytes) 000109[0] 1773668691.584864 mapread 0x779000 thru 0x7832be (0xa2bf bytes) 000110[1] 1773668691.667150 mapread 0x84d000 thru 0x850ca6 (0x3ca7 bytes) 000114[0] 1773668691.708717 mapwrite 0x051000 thru 0x05d4bf (0xc4c0 bytes) 000119[1] 1773668691.902810 trunc from 0x87103a to 0x076c15 (0x7fa424 bytes) 000122[0] 1773668691.985131 write_OD 0x800000 thru 0x809fff (0xa000 bytes) 000125[0] 1773668692.097007 mapread 0x734000 thru 0x739f0d (0x5f0e bytes) 000127[0] 1773668692.144604 mapread 0x695000 thru 0x695359 (0x35a bytes) 000128[1] 1773668692.159551 read 0x4b4000 thru 0x4badfb (0x6dfc bytes) 000129[0] 1773668692.177198 read_OD 0x3e0000 thru 0x3e5fff (0x6000 bytes) 000131[1] 1773668692.186576 write 0x5d0000 thru 0x5d66ff (0x6700 bytes) 000132[1] 1773668692.227732 read 0x6b2000 thru 0x6b8e64 (0x6e65 bytes) 000142[1] 1773668692.408671 mapread 0x400000 thru 0x402ec9 (0x2eca bytes) 000143[1] 1773668692.449647 read 0x559000 thru 0x55c2ca (0x32cb bytes) 000145[0] 1773668692.477361 trunc from 0x80a000 to 0x779949 (0x906b6 bytes) 000147[0] 1773668692.580715 mapread 0x295000 thru 0x2a1e54 (0xce55 bytes) 000152[1] 1773668692.658352 mapwrite 0x45a000 thru 0x45d514 (0x3515 bytes) 000153[1] 1773668692.767057 write 0x9c7000 thru 0x9cbe05 (0x4e06 bytes) 000154[1] 1773668692.783306 mapwrite 0x946000 thru 0x94f9bd (0x99be bytes) 000155[0] 1773668692.867398 mapwrite 0x5e5000 thru 0x5e9203 (0x4204 bytes) 000156[1] 1773668692.954984 mapwrite 0x0d9000 thru 0x0de7bd (0x57be bytes) 000161[1] 1773668693.096292 trunc from 0x9cbe06 to 0x0b3a52 (0x9183b3 bytes) 000162[0] 1773668693.130214 mapwrite 0x0c9000 thru 0x0d6621 (0xd622 bytes) 000163[0] 1773668693.291121 trunc from 0x0d6622 to 0x76a8db (0x6942ba bytes) 000165[0] 1773668693.318496 write_OD 0x508000 thru 0x514fff (0xd000 bytes) 000166[0] 1773668693.397746 trunc from 0x76a8dc to 0x1c66b4 (0x5a4227 bytes) 000167[1] 1773668693.436441 mapwrite 0x246000 thru 0x246e16 (0xe17 bytes) 000168[0] 1773668693.545792 trunc from 0x246e17 to 0x7ee75b (0x5a7945 bytes) 000170[1] 1773668693.579576 write 0x8e0000 thru 0x8e5519 (0x551a bytes) 000173[0] 1773668693.627858 write_OD 0x09e000 thru 0x0acfff (0xf000 bytes) 000175[1] 1773668693.684564 read 0x212000 thru 0x220ce0 (0xece1 bytes) 000178[1] 1773668693.698095 read 0x3bc000 thru 0x3c9a6b (0xda6c bytes) 000180[1] 1773668693.721267 mapwrite 0x4b1000 thru 0x4b6ae5 (0x5ae6 bytes) 000183[0] 1773668693.836932 mapread 0x897000 thru 0x8a35fa (0xc5fb bytes) 000184[0] 1773668693.893457 mapwrite 0x72e000 thru 0x7393a2 (0xb3a3 bytes) 000186[1] 1773668693.993503 mapread 0x1a6000 thru 0x1b476b (0xe76c bytes) 000187[1] 1773668694.071748 mapwrite 0x57e000 thru 0x58a4fd (0xc4fe bytes) 000190[0] 1773668694.218983 read_OD 0x61f000 thru 0x627fff (0x9000 bytes) 000191[0] 1773668694.244154 mapread 0x469000 thru 0x470de8 (0x7de9 bytes) 000195[1] 1773668694.274942 mapwrite 0x1d7000 thru 0x1dec43 (0x7c44 bytes) 000199[0] 1773668694.425218 write_OD 0x7f4000 thru 0x7f6fff (0x3000 bytes) 000200[0] 1773668694.477930 mapread 0x62e000 thru 0x62f542 (0x1543 bytes) 000203[1] 1773668694.491367 trunc from 0x8e551a to 0x002240 (0x8e32d9 bytes) 000204[1] 1773668694.529119 write 0x3b1000 thru 0x3b4e17 (0x3e18 bytes) 000206[0] 1773668694.562208 trunc from 0x3b4e18 to 0x1aee7b (0x205f9c bytes) 000208[1] 1773668694.615164 read 0x0bc000 thru 0x0be524 (0x2525 bytes) 000209[1] 1773668694.663987 mapread 0x094000 thru 0x094a02 (0xa03 bytes) 000212[1] 1773668694.673546 write 0x72d000 thru 0x73b676 (0xe677 bytes) 000214[0] 1773668694.727433 mapwrite 0x345000 thru 0x353ede (0xeedf bytes) 000223[0] 1773668694.969477 write_OD 0x06d000 thru 0x077fff (0xb000 bytes) 000226[0] 1773668695.048893 read_OD 0x181000 thru 0x189fff (0x9000 bytes) 000229[0] 1773668695.093415 trunc from 0x73b677 to 0x5f393c (0x147d3a bytes) 000230[0] 1773668695.146397 read_OD 0x320000 thru 0x323fff (0x4000 bytes) 000231[0] 1773668695.178455 trunc from 0x5f393d to 0x798895 (0x1a4f59 bytes) 000233[1] 1773668695.235704 read 0x14d000 thru 0x14d8c6 (0x8c7 bytes) 000234[0] 1773668695.262052 read_OD 0x01d000 thru 0x025fff (0x9000 bytes) 000236[0] 1773668695.277018 read_OD 0x570000 thru 0x572fff (0x3000 bytes) 000237[1] 1773668695.284707 read 0x68c000 thru 0x68ec0a (0x2c0b bytes) 000238[1] 1773668695.293100 mapread 0x45f000 thru 0x46401b (0x501c bytes) 000240[1] 1773668695.316093 read 0x37b000 thru 0x3887ca (0xd7cb bytes) 000241[1] 1773668695.325988 write 0x27f000 thru 0x282239 (0x323a bytes) 000245[1] 1773668695.368679 write 0x422000 thru 0x42b7f7 (0x97f8 bytes) 000246[1] 1773668695.376033 mapwrite 0x0b5000 thru 0x0c14ad (0xc4ae bytes) 000250[0] 1773668695.549295 write_OD 0x0a4000 thru 0x0aefff (0xb000 bytes) 000251[0] 1773668695.614760 write_OD 0x6ee000 thru 0x6f5fff (0x8000 bytes) 000256[0] 1773668695.679793 trunc from 0x798896 to 0x1dc31c (0x5bc579 bytes) 000257[1] 1773668695.714133 write_OD 0x302000 thru 0x305fff (0x4000 bytes) 000258[0] 1773668695.751632 read_OD 0x079000 thru 0x086fff (0xe000 bytes) 000259[0] 1773668695.765872 mapwrite 0x904000 thru 0x910d8b (0xcd8c bytes) 000260[0] 1773668696.090341 trunc from 0x910d8c to 0x4afae1 (0x4612aa bytes) 000262[0] 1773668696.125078 trunc from 0x4afae2 to 0x2374cc (0x278615 bytes) 000263[0] 1773668696.168519 read_OD 0x07b000 thru 0x07bfff (0x1000 bytes) 000267[0] 1773668696.213934 write_OD 0x90e000 thru 0x90ffff (0x2000 bytes) 000268[1] 1773668696.269519 mapread 0x5ae000 thru 0x5b866e (0xa66f bytes) 000271[0] 1773668696.306992 mapwrite 0x4be000 thru 0x4c0eab (0x2eac bytes) 000272[1] 1773668696.504825 trunc from 0x910000 to 0x283875 (0x68c78a bytes) 000273[0] 1773668696.565666 write_OD 0x1b5000 thru 0x1bbfff (0x7000 bytes) 000274[1] 1773668696.616677 trunc from 0x283876 to 0x922a12 (0x69f19d bytes) 000275[1] 1773668696.652520 read_OD 0x022000 thru 0x02efff (0xd000 bytes) 000276[1] 1773668696.680331 mapread 0x2f0000 thru 0x2fc2c6 (0xc2c7 bytes) 000277[0] 1773668696.709555 write_OD 0x651000 thru 0x65bfff (0xb000 bytes) 000278[0] 1773668696.768076 trunc from 0x922a13 to 0x858404 (0xca60e bytes) 000280[0] 1773668696.806059 write_OD 0x84e000 thru 0x85bfff (0xe000 bytes) 000281[1] 1773668696.857638 mapwrite 0x828000 thru 0x82b18b (0x318c bytes) 000282[1] 1773668697.003005 trunc from 0x85c000 to 0x5021f6 (0x359e09 bytes) 000283[0] 1773668697.036015 mapread 0x3c8000 thru 0x3ce710 (0x6711 bytes) 000284[0] 1773668697.135272 mapread 0x4a9000 thru 0x4b4293 (0xb294 bytes) 000285[0] 1773668697.226833 mapread 0x398000 thru 0x398700 (0x701 bytes) 000286[0] 1773668697.234046 write_OD 0x8cf000 thru 0x8d4fff (0x6000 bytes) 000287[0] 1773668697.303278 trunc from 0x8d5000 to 0x263ea4 (0x67115b bytes) 000288[1] 1773668697.341875 read_OD 0x062000 thru 0x066fff (0x5000 bytes) 000295[0] 1773668697.392429 mapread 0x088000 thru 0x08cdb2 (0x4db3 bytes) 000302[1] 1773668697.433149 write_OD 0x95e000 thru 0x96afff (0xd000 bytes) 000303[1] 1773668697.507253 mapread 0x2a8000 thru 0x2b5c6c (0xdc6d bytes) 000304[1] 1773668697.549312 trunc from 0x96b000 to 0x580a18 (0x3ea5e7 bytes) 000305[0] 1773668697.581868 read_OD 0x1ce000 thru 0x1dbfff (0xe000 bytes) 000308[0] 1773668697.605618 trunc from 0x580a19 to 0x175a3a (0x40afde bytes) 000310[0] 1773668697.643021 trunc from 0x175a3b to 0x54834a (0x3d2910 bytes) 000311[1] 1773668697.676998 write_OD 0x411000 thru 0x415fff (0x5000 bytes) 000312[0] 1773668697.731858 write_OD 0x2c3000 thru 0x2c9fff (0x7000 bytes) 000313[1] 1773668697.795616 mapread 0x1a6000 thru 0x1b004a (0xa04b bytes) 000314[1] 1773668697.837692 mapread 00000000 thru 0x00fd2e (0xfd2f bytes) 000315[1] 1773668697.873058 read_OD 0x0a2000 thru 0x0adfff (0xc000 bytes) 000316[1] 1773668697.883794 read_OD 0x52c000 thru 0x532fff (0x7000 bytes) 000317[0] 1773668697.894961 write_OD 0x5f7000 thru 0x5fafff (0x4000 bytes) 000318[1] 1773668697.990669 trunc from 0x5fb000 to 0x50d27b (0xedd84 bytes) 000319[1] 1773668698.061436 mapwrite 0x48a000 thru 0x48f42a (0x542b bytes) 000321[1] 1773668698.144977 mapwrite 0x1ba000 thru 0x1be737 (0x4738 bytes) 000326[1] 1773668698.286941 mapread 0x156000 thru 0x1560e6 (0xe7 bytes) 000328[1] 1773668698.323181 write_OD 0x111000 thru 0x117fff (0x7000 bytes) 000329[1] 1773668698.371857 write_OD 0x491000 thru 0x491fff (0x1000 bytes) 000330[0] 1773668698.435263 trunc from 0x50d27c to 0x0e2c81 (0x42a5fa bytes) 000332[0] 1773668698.480536 write_OD 0x55e000 thru 0x55ffff (0x2000 bytes) 000336[1] 1773668698.525991 write_OD 0x468000 thru 0x470fff (0x9000 bytes) 000338[1] 1773668698.592143 mapread 0x315000 thru 0x318703 (0x3704 bytes) 000339[0] 1773668698.637217 read_OD 0x3ff000 thru 0x407fff (0x9000 bytes) 000340[0] 1773668698.663802 mapwrite 0x960000 thru 0x96bc36 (0xbc37 bytes) 000342[1] 1773668698.815427 mapread 0x3eb000 thru 0x3f6ca5 (0xbca6 bytes) 000344[0] 1773668698.890255 trunc from 0x96bc37 to 0x404f04 (0x566d32 bytes) 000345[1] 1773668698.935394 trunc from 0x404f05 to 0x948e21 (0x543f1d bytes) 000346[0] 1773668698.965523 mapread 0x221000 thru 0x227604 (0x6605 bytes) 000347[0] 1773668698.994001 read_OD 0x46f000 thru 0x47dfff (0xf000 bytes) 000350[0] 1773668699.026361 mapread 0x4b2000 thru 0x4bbefd (0x9efe bytes) 000352[1] 1773668699.060557 mapread 0x085000 thru 0x0885de (0x35df bytes) 000354[0] 1773668699.111676 mapread 0x65c000 thru 0x66917b (0xd17c bytes) 000357[1] 1773668699.169135 mapread 0x2c8000 thru 0x2d2506 (0xa507 bytes) 000363[1] 1773668699.215946 write_OD 0x098000 thru 0x09ffff (0x8000 bytes) 000365[1] 1773668699.274272 write_OD 0x1b3000 thru 0x1befff (0xc000 bytes) 000368[0] 1773668699.321971 trunc from 0x948e22 to 0x0010f9 (0x947d28 bytes) 000370[1] 1773668699.375964 write_OD 0x81c000 thru 0x82afff (0xf000 bytes) 000372[1] 1773668699.441677 trunc from 0x82b000 to 0x3d796b (0x453694 bytes) 000377[0] 1773668699.525335 mapwrite 0x95c000 thru 0x967928 (0xb929 bytes) 000378[1] 1773668699.671723 write_OD 0x217000 thru 0x217fff (0x1000 bytes) 000380[1] 1773668699.729024 mapwrite 0x724000 thru 0x725a9f (0x1aa0 bytes) 000382[1] 1773668699.817479 read_OD 0x700000 thru 0x708fff (0x9000 bytes) 000388[0] 1773668699.850193 write_OD 0x4ab000 thru 0x4b9fff (0xf000 bytes) 000391[1] 1773668699.921476 read_OD 0x7c9000 thru 0x7d5fff (0xd000 bytes) 000392[1] 1773668699.941808 write_OD 0x109000 thru 0x113fff (0xb000 bytes) 000393[0] 1773668700.015856 mapwrite 0x1ea000 thru 0x1f310e (0x910f bytes) 000396[1] 1773668700.089560 read_OD 0x79a000 thru 0x79ffff (0x6000 bytes) 000397[1] 1773668700.108705 read_OD 0x2c0000 thru 0x2cbfff (0xc000 bytes) 000398[0] 1773668700.115740 trunc from 0x967929 to 0x1c8134 (0x79f7f4 bytes) 000400[1] 1773668700.183964 trunc from 0x1c8135 to 0x6bf0d3 (0x4f6f9f bytes) 000402[0] 1773668700.259255 mapwrite 0x965000 thru 0x96ae9a (0x5e9b bytes) 000404[1] 1773668700.400945 write_OD 0x175000 thru 0x17ffff (0xb000 bytes) 000408[1] 1773668700.469299 mapwrite 0x0ff000 thru 0x10725c (0x825d bytes) 000409[0] 1773668700.563978 read_OD 0x210000 thru 0x213fff (0x4000 bytes) 000410[0] 1773668700.594597 mapread 0x174000 thru 0x17fe51 (0xbe52 bytes) 000411[0] 1773668700.629804 write_OD 0x0ff000 thru 0x105fff (0x7000 bytes) 000415[1] 1773668700.692505 mapwrite 0x305000 thru 0x30b987 (0x6988 bytes) 000416[0] 1773668700.829411 write_OD 0x044000 thru 0x04efff (0xb000 bytes) 000417[0] 1773668700.918618 trunc from 0x96ae9b to 0x97baf6 (0x10c5c bytes) 000418[0] 1773668700.957411 mapread 0x31a000 thru 0x31a89c (0x89d bytes) 000420[0] 1773668700.999204 mapread 0x84b000 thru 0x85300b (0x800c bytes) 000421[0] 1773668701.023442 write_OD 0x581000 thru 0x58cfff (0xc000 bytes) 000424[1] 1773668701.089576 mapread 0x595000 thru 0x59bcda (0x6cdb bytes) 000428[1] 1773668701.120496 trunc from 0x97baf7 to 0x00d83f (0x96e2b7 bytes) 000434[0] 1773668701.278659 read_OD 0x001000 thru 0x001fff (0x1000 bytes) 000435[1] 1773668701.318715 mapwrite 0x56c000 thru 0x57b62f (0xf630 bytes) 000436[0] 1773668701.468475 read_OD 0x0f0000 thru 0x0fafff (0xb000 bytes) 000439[0] 1773668701.501166 mapwrite 0x86a000 thru 0x86b6fd (0x16fe bytes) 000440[1] 1773668701.616093 mapread 0x36a000 thru 0x37733b (0xd33c bytes) 000441[1] 1773668701.689129 write_OD 0x704000 thru 0x708fff (0x5000 bytes) 000442[1] 1773668701.758766 read_OD 0x41b000 thru 0x422fff (0x8000 bytes) 000444[0] 1773668701.772098 read_OD 0x19a000 thru 0x19ffff (0x6000 bytes) 000447[0] 1773668701.791648 trunc from 0x86b6fe to 0x41167d (0x45a080 bytes) 000448[1] 1773668701.844795 write_OD 0x91d000 thru 0x925fff (0x9000 bytes) 000450[0] 1773668701.912523 mapwrite 0x3e7000 thru 0x3f3544 (0xc545 bytes) 000453[1] 1773668702.064471 mapread 0x022000 thru 0x023bc3 (0x1bc4 bytes) 000458[0] 1773668702.103682 mapread 0x71e000 thru 0x71f717 (0x1718 bytes) 000460[1] 1773668702.120415 mapread 0x524000 thru 0x52c9eb (0x89ec bytes) 000464[0] 1773668702.171839 trunc from 0x926000 to 0x548096 (0x3ddf69 bytes) 000465[1] 1773668702.239943 read_OD 0x4ed000 thru 0x4f8fff (0xc000 bytes) 000466[0] 1773668702.276898 trunc from 0x548097 to 0x77eb32 (0x236a9c bytes) 000470[1] 1773668702.316671 mapwrite 0x3a6000 thru 0x3ad2a9 (0x72aa bytes) 000473[0] 1773668702.413032 trunc from 0x77eb33 to 0x6bfd4c (0xbede6 bytes) 000476[0] 1773668702.493809 trunc from 0x6bfd4d to 0x541d6c (0x17dfe0 bytes) 000477[1] 1773668702.536780 write_OD 0x86e000 thru 0x878fff (0xb000 bytes) 000478[1] 1773668702.610391 trunc from 0x879000 to 0x629cf9 (0x24f306 bytes) 000479[0] 1773668702.662737 read_OD 0x24e000 thru 0x254fff (0x7000 bytes) 000480[0] 1773668702.712711 write_OD 0x103000 thru 0x110fff (0xe000 bytes) 000487[1] 1773668702.802722 mapwrite 0x5fe000 thru 0x60398d (0x598e bytes) 000489[0] 1773668702.893998 mapread 0x056000 thru 0x061221 (0xb222 bytes) 000491[0] 1773668702.934839 trunc from 0x629cfa to 0x06383c (0x5c64bd bytes) 000492[0] 1773668702.969598 read_OD 0x02f000 thru 0x030fff (0x2000 bytes) 000495[1] 1773668703.002646 read_OD 0x002000 thru 0x005fff (0x4000 bytes) All operations completed A-OK! PASS 16a (42s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:45:07 (1773668707) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.319299 s, 26.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.249987 s, 33.6 MB/s Chance of close/open is 1 in 50 Seed set to 8709 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1773668710.131082 mapwrite 0x7977e2 thru 0x7994e4 (0x1d03 bytes) 000003[1] 1773668710.268681 mapwrite 0x80d895 thru 0x80db33 (0x29f bytes) 000005[1] 1773668710.462860 mapread 0x270229 thru 0x27182b (0x1603 bytes) 000008[1] 1773668710.517706 read 0x760445 thru 0x76094a (0x506 bytes) 000012[1] 1773668710.552123 mapwrite 0x13ce8d thru 0x13deb3 (0x1027 bytes) 000013[1] 1773668710.631651 mapwrite 0x573508 thru 0x573787 (0x280 bytes) 000014[1] 1773668710.674984 mapwrite 0x3295b2 thru 0x32a396 (0xde5 bytes) 000015[1] 1773668710.765479 write 0x5c5861 thru 0x5c687d (0x101d bytes) 000016[0] 1773668710.809142 write 0x338aa2 thru 0x33a89a (0x1df9 bytes) 000020[1] 1773668710.904639 read 0x142001 thru 0x143341 (0x1341 bytes) 000023[1] 1773668710.991264 write 0x37774d thru 0x377cad (0x561 bytes) 000025[0] 1773668711.008998 read 0x37393e thru 0x374c25 (0x12e8 bytes) truncating to largest ever: 0x15a74c 000028[1] 1773668711.083683 trunc from 0x80db34 to 0x15a74b (0x6b33e8 bytes) 000032[0] 1773668711.204353 write 0x677f98 thru 0x679c19 (0x1c82 bytes) 000035[0] 1773668711.304640 write 0x547b77 thru 0x5490dc (0x1566 bytes) 000036[0] 1773668711.334644 mapread 0x39841c thru 0x399165 (0xd4a bytes) 000037[1] 1773668711.409199 mapread 0x4e6eed thru 0x4e73bb (0x4cf bytes) truncating to largest ever: 0x978445 000040[1] 1773668711.460762 trunc from 0x679c1a to 0x978444 (0x2fe82b bytes) 000041[0] 1773668711.544801 read 0x7a392a thru 0x7a412e (0x805 bytes) 000042[0] 1773668711.575779 mapread 0x0129d9 thru 0x014108 (0x1730 bytes) 000044[1] 1773668711.617600 trunc from 0x978445 to 0x43f1fc (0x539248 bytes) 000049[0] 1773668711.737199 mapread 0x277d56 thru 0x2781a8 (0x453 bytes) 000052[0] 1773668711.830779 mapwrite 0x1d847b thru 0x1d8568 (0xee bytes) 000053[1] 1773668711.945151 write 0x51ab3a thru 0x51ae62 (0x329 bytes) 000057[0] 1773668712.026539 write 0x602c1a thru 0x604699 (0x1a80 bytes) 000060[1] 1773668712.075721 read 0x48fc0d thru 0x491980 (0x1d74 bytes) 000061[1] 1773668712.112882 mapread 0x2b3d8c thru 0x2b43f6 (0x66b bytes) 000063[1] 1773668712.149418 write 0x56f531 thru 0x56f5ff (0xcf bytes) 000066[0] 1773668712.269120 write 0x935b31 thru 0x936f6c (0x143c bytes) 000067[0] 1773668712.334079 read 0x527bfb thru 0x5299d8 (0x1dde bytes) 000068[0] 1773668712.388515 trunc from 0x936f6d to 0x67a460 (0x2bcb0c bytes) 000069[0] 1773668712.441336 write 0x31a234 thru 0x31bfd9 (0x1da6 bytes) 000070[0] 1773668712.455814 mapwrite 0x8f9658 thru 0x8fb256 (0x1bff bytes) 000072[0] 1773668712.574089 mapwrite 0x92a4c7 thru 0x92a7ae (0x2e8 bytes) 000079[0] 1773668712.763169 read 0x1ce094 thru 0x1ce993 (0x900 bytes) 000080[0] 1773668712.769739 write 0x167bcf thru 0x16940c (0x183e bytes) 000086[0] 1773668712.889489 write 0x1f9fc9 thru 0x1faf90 (0xfc8 bytes) 000087[0] 1773668712.932003 read 0x1d9939 thru 0x1da07b (0x743 bytes) 000089[1] 1773668713.011360 write 0x91ccab thru 0x91ec5d (0x1fb3 bytes) 000091[0] 1773668713.077543 trunc from 0x92a7af to 0x07f19f (0x8ab60f bytes) 000092[0] 1773668713.123641 mapwrite 0x12d65e thru 0x12f5a3 (0x1f46 bytes) 000095[0] 1773668713.257608 mapwrite 0x24e10e thru 0x24fdf9 (0x1cec bytes) 000097[0] 1773668713.413101 trunc from 0x24fdfa to 0x768719 (0x518920 bytes) 000098[0] 1773668713.453451 trunc from 0x76871a to 0x8ecd03 (0x1845ea bytes) 000099[1] 1773668713.521043 read 0x234cb2 thru 0x2362a4 (0x15f3 bytes) 000104[1] 1773668713.603951 read 0x1e311a thru 0x1e38ac (0x793 bytes) 000105[1] 1773668713.628285 write 0x21126f thru 0x211f68 (0xcfa bytes) 000110[0] 1773668713.764857 write 0x13c225 thru 0x13dcee (0x1aca bytes) 000111[0] 1773668713.792957 mapread 0x51019d thru 0x5120cc (0x1f30 bytes) 000115[1] 1773668713.817078 read 0x04d03b thru 0x04ed54 (0x1d1a bytes) 000116[0] 1773668713.870703 mapwrite 0x5f6c2a thru 0x5f89a4 (0x1d7b bytes) 000117[0] 1773668713.975123 write 0x720204 thru 0x7216e8 (0x14e5 bytes) 000119[1] 1773668714.012432 mapwrite 0x78f34c thru 0x78f738 (0x3ed bytes) 000120[0] 1773668714.123635 write 0x6ced47 thru 0x6cfc4c (0xf06 bytes) 000123[0] 1773668714.199656 write 0x0752e1 thru 0x076930 (0x1650 bytes) 000126[1] 1773668714.342345 mapread 0x1a8947 thru 0x1aa491 (0x1b4b bytes) 000127[1] 1773668714.429944 mapwrite 0x4ff69f thru 0x500094 (0x9f6 bytes) 000129[1] 1773668714.581612 write 0x7198ad thru 0x71b7ba (0x1f0e bytes) 000130[0] 1773668714.597706 write 0x23ff28 thru 0x240465 (0x53e bytes) 000131[1] 1773668714.644774 mapwrite 0x140d54 thru 0x1422ff (0x15ac bytes) 000132[0] 1773668714.738727 mapread 0x8339dd thru 0x83577b (0x1d9f bytes) 000138[1] 1773668714.846948 read 0x06b65b thru 0x06b918 (0x2be bytes) 000140[0] 1773668714.885984 trunc from 0x8ecd04 to 0x61853c (0x2d47c7 bytes) 000142[0] 1773668714.931423 write 0x045048 thru 0x045675 (0x62e bytes) 000146[0] 1773668715.005320 mapread 0x1befa4 thru 0x1bf613 (0x670 bytes) 000151[0] 1773668715.117123 trunc from 0x61853d to 0x59835b (0x801e1 bytes) 000152[1] 1773668715.182522 mapwrite 0x4d1420 thru 0x4d169b (0x27c bytes) 000153[1] 1773668715.300178 trunc from 0x59835c to 0x219cc2 (0x37e699 bytes) 000154[1] 1773668715.359536 write 0x0667dc thru 0x068766 (0x1f8b bytes) 000160[1] 1773668715.468135 trunc from 0x219cc3 to 0x955dd5 (0x73c113 bytes) 000161[1] 1773668715.514655 mapwrite 0x164925 thru 0x165b87 (0x1263 bytes) 000163[0] 1773668715.623426 mapwrite 0x4ddb4a thru 0x4def51 (0x1408 bytes) 000165[0] 1773668715.747036 mapwrite 0x0c4ba2 thru 0x0c5f3f (0x139e bytes) 000167[1] 1773668715.815298 mapwrite 0x0cd42e thru 0x0cddb3 (0x986 bytes) 000168[1] 1773668715.925032 write 0x25bcd9 thru 0x25c2f5 (0x61d bytes) 000169[0] 1773668715.962467 write 0x95758e thru 0x95839f (0xe12 bytes) 000170[1] 1773668715.972433 mapread 0x3046f9 thru 0x305ed5 (0x17dd bytes) 000172[1] 1773668716.039754 read 0x25a017 thru 0x25b212 (0x11fc bytes) 000173[1] 1773668716.061363 mapwrite 0x7b862e thru 0x7ba48e (0x1e61 bytes) 000175[1] 1773668716.211630 write 0x5d6429 thru 0x5d7350 (0xf28 bytes) 000176[0] 1773668716.227688 trunc from 0x9583a0 to 0x39280d (0x5c5b92 bytes) 000177[1] 1773668716.342578 mapwrite 0x4ccc8e thru 0x4cd6f4 (0xa67 bytes) 000179[0] 1773668716.485582 write 0x429e35 thru 0x42a7e9 (0x9b5 bytes) 000180[1] 1773668716.537767 mapwrite 0x2d9c44 thru 0x2da6ad (0xa6a bytes) 000181[0] 1773668716.611258 mapread 0x4ca63a thru 0x4cbe19 (0x17e0 bytes) 000184[0] 1773668716.662360 mapread 0x396e75 thru 0x397fa5 (0x1131 bytes) 000187[1] 1773668716.693650 mapwrite 0x1c36b4 thru 0x1c4743 (0x1090 bytes) 000193[1] 1773668716.868815 trunc from 0x4cd6f5 to 0x223cb3 (0x2a9a41 bytes) 000195[1] 1773668716.904684 write 0x09684a thru 0x097d4d (0x1504 bytes) 000198[0] 1773668716.929603 trunc from 0x223cb4 to 0x2fb5f3 (0xd7940 bytes) 000199[0] 1773668716.967784 write 0x4e3b84 thru 0x4e4c23 (0x10a0 bytes) 000200[0] 1773668716.969562 trunc from 0x4e4c24 to 0x2718a4 (0x27337f bytes) 000202[1] 1773668717.037362 read 0x1d58c3 thru 0x1d7795 (0x1ed3 bytes) 000203[0] 1773668717.079123 mapwrite 0x8047cc thru 0x80500b (0x840 bytes) 000206[1] 1773668717.304066 mapwrite 0x9efbd8 thru 0x9f165e (0x1a87 bytes) 000207[1] 1773668717.465977 write 0x9edcc2 thru 0x9ee7d6 (0xb15 bytes) 000209[0] 1773668717.505515 trunc from 0x9f165f to 0x5a1ac3 (0x44fb9b bytes) 000211[1] 1773668717.591194 write 0x577988 thru 0x578c49 (0x12c2 bytes) 000212[0] 1773668717.637261 write 0x2559f9 thru 0x255be3 (0x1eb bytes) 000217[0] 1773668717.751787 mapwrite 0x9e4e73 thru 0x9e662a (0x17b8 bytes) 000219[1] 1773668717.879245 write 0x998604 thru 0x99a45f (0x1e5c bytes) 000220[1] 1773668717.925675 read 0x432fa4 thru 0x434a51 (0x1aae bytes) 000221[1] 1773668717.967251 mapwrite 0x506e85 thru 0x508328 (0x14a4 bytes) 000224[1] 1773668718.168908 trunc from 0x9e662b to 0x37b5a3 (0x66b087 bytes) truncating to largest ever: 0x9b3e53 000225[0] 1773668718.206532 trunc from 0x37b5a4 to 0x9b3e52 (0x6388af bytes) 000227[1] 1773668718.259085 read 0x75f0e3 thru 0x75f657 (0x575 bytes) 000229[0] 1773668718.295002 write 0x424db8 thru 0x4262c4 (0x150d bytes) 000230[0] 1773668718.315907 mapread 0x0d91ed thru 0x0da271 (0x1085 bytes) 000231[1] 1773668718.324326 read 0x51e5a0 thru 0x520114 (0x1b75 bytes) 000233[1] 1773668718.360221 mapwrite 0x3f4806 thru 0x3f6009 (0x1804 bytes) 000237[1] 1773668718.479108 mapwrite 0x6f40af thru 0x6f46be (0x610 bytes) 000239[0] 1773668718.561131 mapwrite 0x31e676 thru 0x31eee7 (0x872 bytes) 000240[0] 1773668718.651688 read 0x2a5fe6 thru 0x2a7cc4 (0x1cdf bytes) 000241[1] 1773668718.675505 mapread 0x92c65e thru 0x92ce7c (0x81f bytes) 000243[0] 1773668718.733159 write 0x9d2cd9 thru 0x9d3ff5 (0x131d bytes) 000245[0] 1773668718.801325 mapwrite 0x50beb7 thru 0x50d84c (0x1996 bytes) 000246[1] 1773668718.910719 mapwrite 0x497ccc thru 0x499594 (0x18c9 bytes) 000248[1] 1773668719.057518 mapwrite 0x63b3a8 thru 0x63bac2 (0x71b bytes) 000250[0] 1773668719.209310 mapwrite 0x92ed4c thru 0x930bbf (0x1e74 bytes) 000251[1] 1773668719.349656 mapwrite 0x85d456 thru 0x85e682 (0x122d bytes) 000252[0] 1773668719.454796 trunc from 0x9d3ff6 to 0x5b8841 (0x41b7b4 bytes) 000253[0] 1773668719.504404 mapwrite 0x360c1c thru 0x361f26 (0x130b bytes) truncating to largest ever: 0x9babdb 000254[1] 1773668719.567585 trunc from 0x5b8842 to 0x9babda (0x402399 bytes) 000258[1] 1773668719.720212 mapwrite 0x20daec thru 0x20f80c (0x1d21 bytes) 000259[1] 1773668719.838452 mapread 0x2368b7 thru 0x237103 (0x84d bytes) 000261[0] 1773668719.905270 trunc from 0x9babdb to 0x464124 (0x556ab6 bytes) 000262[1] 1773668719.999031 mapread 0x222fbf thru 0x2239ba (0x9fc bytes) 000264[0] 1773668720.065039 mapread 0x45732b thru 0x4573c0 (0x96 bytes) 000266[0] 1773668720.083074 read 0x0f3d3f thru 0x0f5d2b (0x1fed bytes) 000269[1] 1773668720.150844 mapwrite 0x2b928d thru 0x2baa82 (0x17f6 bytes) 000270[1] 1773668720.248970 trunc from 0x464125 to 0x2e2da2 (0x181382 bytes) 000272[1] 1773668720.344728 read 0x077ea3 thru 0x07960f (0x176d bytes) 000273[1] 1773668720.442988 read 0x1910f5 thru 0x19127d (0x189 bytes) 000274[1] 1773668720.471719 read 0x1743d7 thru 0x1752bb (0xee5 bytes) 000275[1] 1773668720.497294 mapread 0x26d1a6 thru 0x26d7ca (0x625 bytes) 000276[0] 1773668720.522612 read 0x05d809 thru 0x05d99f (0x197 bytes) 000279[0] 1773668720.637746 mapread 0x2961d1 thru 0x298188 (0x1fb8 bytes) 000282[0] 1773668720.723296 mapread 0x0110cd thru 0x01278f (0x16c3 bytes) 000284[1] 1773668720.780567 write 0x9df44b thru 0x9e06d9 (0x128f bytes) 000285[1] 1773668720.845955 read 0x01999b thru 0x01a95d (0xfc3 bytes) 000288[0] 1773668720.884364 read 0x9082ec thru 0x9090ac (0xdc1 bytes) 000292[0] 1773668720.943656 write 0x256bb7 thru 0x257a1f (0xe69 bytes) 000293[0] 1773668720.971899 write 0x92e45b thru 0x9301f2 (0x1d98 bytes) 000296[1] 1773668721.027350 trunc from 0x9e06da to 0x19dd4b (0x84298e bytes) 000298[0] 1773668721.116742 mapwrite 0x76cc86 thru 0x76eaf2 (0x1e6d bytes) 000299[1] 1773668721.276834 write 0x764976 thru 0x764df4 (0x47f bytes) 000302[1] 1773668721.355714 write 0x5eb687 thru 0x5ed2d9 (0x1c53 bytes) 000303[1] 1773668721.384156 mapread 0x6f4a38 thru 0x6f568b (0xc54 bytes) 000305[0] 1773668721.436916 read 0x60ed1f thru 0x60f5b9 (0x89b bytes) 000307[1] 1773668721.492355 write 0x3450b5 thru 0x34690b (0x1857 bytes) 000310[0] 1773668721.529439 write 0x92a4b2 thru 0x92a94e (0x49d bytes) 000311[0] 1773668721.547700 trunc from 0x92a94f to 0x67baba (0x2aee94 bytes) 000312[1] 1773668721.613506 mapwrite 0x3d4ae9 thru 0x3d5a7a (0xf92 bytes) 000313[0] 1773668721.708041 trunc from 0x67babb to 0x55d4b9 (0x11e601 bytes) 000314[1] 1773668721.738467 trunc from 0x55d4ba to 0x74c683 (0x1ef1ca bytes) 000317[1] 1773668721.831289 mapwrite 0x40a806 thru 0x40c3fa (0x1bf5 bytes) 000320[1] 1773668721.901720 mapwrite 0x88d546 thru 0x88d85f (0x31a bytes) 000321[0] 1773668721.962010 read 0x144590 thru 0x1457c5 (0x1236 bytes) 000322[0] 1773668721.987561 mapwrite 0x50520c thru 0x506c98 (0x1a8d bytes) 000323[1] 1773668722.065744 mapwrite 0x3592ac thru 0x35ada4 (0x1af9 bytes) 000325[1] 1773668722.146756 mapread 0x03a8f3 thru 0x03c77e (0x1e8c bytes) 000326[1] 1773668722.183514 write 0x77855d thru 0x77a324 (0x1dc8 bytes) 000327[0] 1773668722.195631 write 0x8ce715 thru 0x8d062b (0x1f17 bytes) 000328[0] 1773668722.227426 trunc from 0x8d062c to 0x705987 (0x1caca4 bytes) 000329[0] 1773668722.277584 mapwrite 0x71026e thru 0x711a33 (0x17c6 bytes) 000330[1] 1773668722.388166 trunc from 0x711a34 to 0x610a73 (0x100fc0 bytes) 000332[1] 1773668722.461434 read 0x255816 thru 0x255a8f (0x27a bytes) 000333[1] 1773668722.492565 trunc from 0x610a74 to 0x25bd50 (0x3b4d23 bytes) 000337[0] 1773668722.607143 write 0x77f57d thru 0x77fffa (0xa7e bytes) 000338[0] 1773668722.665822 read 0x0f6000 thru 0x0f6e1d (0xe1e bytes) 000340[1] 1773668722.769634 write 0x634e10 thru 0x635c16 (0xe07 bytes) 000345[1] 1773668722.853579 mapread 0x37e85f thru 0x38015d (0x18ff bytes) 000349[0] 1773668722.948514 read 0x328253 thru 0x3288b7 (0x665 bytes) 000350[0] 1773668722.980052 read 0x06a59c thru 0x06bc37 (0x169c bytes) 000352[1] 1773668723.005754 write 0x1b783e thru 0x1b8872 (0x1035 bytes) 000354[1] 1773668723.061712 mapread 0x6738e2 thru 0x674869 (0xf88 bytes) 000355[1] 1773668723.085223 write 0x1687c0 thru 0x169791 (0xfd2 bytes) 000356[0] 1773668723.096417 mapwrite 0x61b37f thru 0x61b4b9 (0x13b bytes) 000357[0] 1773668723.203042 mapwrite 0x2473a0 thru 0x247fca (0xc2b bytes) 000358[1] 1773668723.295476 mapread 0x6a6432 thru 0x6a7ed9 (0x1aa8 bytes) 000359[0] 1773668723.330261 write 0x2a13b4 thru 0x2a2eef (0x1b3c bytes) 000362[0] 1773668723.365233 trunc from 0x77fffb to 0x1c508e (0x5baf6c bytes) 000365[0] 1773668723.388697 mapwrite 0x181b16 thru 0x182588 (0xa73 bytes) 000366[1] 1773668723.435572 write 0x45498c thru 0x45585c (0xed1 bytes) 000368[1] 1773668723.486795 read 0x3c9c02 thru 0x3ca674 (0xa73 bytes) 000369[1] 1773668723.512984 mapread 0x421854 thru 0x421af6 (0x2a3 bytes) 000371[0] 1773668723.534843 write 0x7fb7a7 thru 0x7fbc4b (0x4a5 bytes) 000372[0] 1773668723.594577 mapread 0x3362de thru 0x337562 (0x1285 bytes) 000373[1] 1773668723.606606 mapread 0x73f7d5 thru 0x7415d9 (0x1e05 bytes) 000375[0] 1773668723.670320 read 0x5b7c46 thru 0x5b888a (0xc45 bytes) 000380[1] 1773668723.679120 write 0x8c8e49 thru 0x8c8fff (0x1b7 bytes) 000381[1] 1773668723.691654 mapread 0x1ffdda thru 0x201b10 (0x1d37 bytes) 000384[0] 1773668723.770973 mapwrite 0x37e6a5 thru 0x3803d1 (0x1d2d bytes) 000386[1] 1773668723.868933 write 0x85026a thru 0x851749 (0x14e0 bytes) 000389[0] 1773668723.937632 mapread 0x066de0 thru 0x06820e (0x142f bytes) 000393[0] 1773668723.979507 mapwrite 0x996877 thru 0x996c6e (0x3f8 bytes) 000394[0] 1773668724.079481 mapwrite 0x9e89ce thru 0x9e98a1 (0xed4 bytes) 000395[0] 1773668724.138568 mapwrite 0x4d831b thru 0x4d8be9 (0x8cf bytes) 000399[0] 1773668724.224338 write 0x75a011 thru 0x75b0dd (0x10cd bytes) 000401[0] 1773668724.311217 read 0x4fb430 thru 0x4fc45a (0x102b bytes) 000402[0] 1773668724.316194 trunc from 0x9e98a2 to 0x29068a (0x759217 bytes) 000404[1] 1773668724.332720 mapread 0x24b665 thru 0x24cf26 (0x18c2 bytes) 000405[0] 1773668724.376289 read 0x008902 thru 0x009c37 (0x1336 bytes) 000408[1] 1773668724.485453 mapwrite 0x650cbd thru 0x651dbf (0x1103 bytes) 000410[1] 1773668724.680363 read 0x4f1e0b thru 0x4f24fd (0x6f3 bytes) 000412[1] 1773668724.732877 mapread 0x59a57b thru 0x59b4da (0xf60 bytes) 000413[0] 1773668724.765221 mapwrite 0x8d871d thru 0x8d8e57 (0x73b bytes) 000414[0] 1773668724.970692 mapread 0x5b3cbd thru 0x5b40c6 (0x40a bytes) 000415[1] 1773668724.991307 mapwrite 0x4f6949 thru 0x4f7a85 (0x113d bytes) 000417[0] 1773668725.144269 read 0x2caf52 thru 0x2cc83f (0x18ee bytes) 000420[0] 1773668725.219042 read 0x7de9d6 thru 0x7debae (0x1d9 bytes) 000422[1] 1773668725.300002 write 0x20df65 thru 0x20ef7a (0x1016 bytes) 000423[1] 1773668725.369087 trunc from 0x8d8e58 to 0x04f26a (0x889bed bytes) 000425[1] 1773668725.429185 mapwrite 0x6ec6ee thru 0x6ec7b0 (0xc3 bytes) 000426[0] 1773668725.510032 mapwrite 0x64a820 thru 0x64b660 (0xe41 bytes) 000430[0] 1773668725.700250 read 0x4913f9 thru 0x491f77 (0xb7f bytes) 000431[0] 1773668725.714856 mapread 0x111cdd thru 0x113851 (0x1b75 bytes) 000432[1] 1773668725.740194 trunc from 0x6ec7b1 to 0x80d03b (0x12088b bytes) 000433[1] 1773668725.786232 trunc from 0x80d03c to 0x1d0a48 (0x63c5f3 bytes) 000434[1] 1773668725.803533 write 0x3ae29c thru 0x3af35d (0x10c2 bytes) 000437[1] 1773668725.843514 write 0x4d819c thru 0x4d8fcf (0xe34 bytes) 000440[1] 1773668725.892080 read 0x165e84 thru 0x167a77 (0x1bf4 bytes) 000441[1] 1773668725.897143 mapread 0x4962a2 thru 0x49697d (0x6dc bytes) 000445[1] 1773668725.979314 mapwrite 0x05840f thru 0x059d07 (0x18f9 bytes) 000449[1] 1773668726.057323 mapwrite 0x1130a6 thru 0x1131e7 (0x142 bytes) 000453[1] 1773668726.183160 trunc from 0x4d8fd0 to 0x3bb853 (0x11d77c bytes) 000455[1] 1773668726.209479 write 0x0aae2a thru 0x0abefd (0x10d4 bytes) 000456[0] 1773668726.218208 trunc from 0x3bb854 to 0x0777ce (0x344085 bytes) 000457[0] 1773668726.263995 mapwrite 0x53ce43 thru 0x53e1d2 (0x1390 bytes) 000459[0] 1773668726.406661 mapread 0x0d43e0 thru 0x0d5f2d (0x1b4e bytes) 000460[0] 1773668726.420642 write 0x4f98e4 thru 0x4fafe9 (0x1706 bytes) 000461[0] 1773668726.426789 read 0x3f7361 thru 0x3f7c3a (0x8da bytes) 000462[0] 1773668726.450650 trunc from 0x53e1d3 to 0x18158a (0x3bcc48 bytes) 000463[1] 1773668726.492517 read 0x064fb0 thru 0x06540c (0x45d bytes) 000465[0] 1773668726.538674 read 0x0d1692 thru 0x0d1a38 (0x3a7 bytes) 000467[0] 1773668726.612038 write 0x10d695 thru 0x10ee1f (0x178b bytes) 000469[0] 1773668726.661276 trunc from 0x18158b to 0x13e16d (0x4341d bytes) 000470[1] 1773668726.684678 mapwrite 0x92f0fb thru 0x92fcd8 (0xbde bytes) 000471[0] 1773668726.812535 mapread 0x6b48f2 thru 0x6b4e43 (0x552 bytes) 000475[1] 1773668726.857511 read 0x17cafb thru 0x17d1d0 (0x6d6 bytes) truncating to largest ever: 0x9dd5ed 000476[1] 1773668726.864018 trunc from 0x92fcd9 to 0x9dd5ec (0xad914 bytes) 000477[1] 1773668726.906632 mapwrite 0x6e6cdd thru 0x6e8777 (0x1a9b bytes) 000480[0] 1773668727.061592 write 0x4f624c thru 0x4f798e (0x1743 bytes) 000481[1] 1773668727.101872 mapread 0x3e04d9 thru 0x3e1452 (0xf7a bytes) 000482[0] 1773668727.137894 read 0x891c61 thru 0x892619 (0x9b9 bytes) 000484[0] 1773668727.178598 trunc from 0x9dd5ed to 0x8ca74d (0x112e9f bytes) 000485[0] 1773668727.246129 mapread 0x460167 thru 0x4614d2 (0x136c bytes) 000490[1] 1773668727.343150 read 0x32ceb7 thru 0x32ebcc (0x1d16 bytes) 000491[0] 1773668727.356665 mapwrite 0x6ee773 thru 0x6efc79 (0x1507 bytes) 000492[1] 1773668727.402949 trunc from 0x8ca74e to 0x8fa70f (0x2ffc2 bytes) 000493[0] 1773668727.449518 mapwrite 0x4cd5de thru 0x4ce244 (0xc67 bytes) 000494[1] 1773668727.529509 mapread 0x02c726 thru 0x02df4b (0x1826 bytes) 000495[1] 1773668727.572907 trunc from 0x8fa710 to 0x0b8232 (0x8424dd bytes) 000496[1] 1773668727.606896 mapwrite 0x9689d4 thru 0x968c01 (0x22e bytes) 000497[0] 1773668727.678566 trunc from 0x968c02 to 0x30a6a8 (0x65e559 bytes) 000499[0] 1773668727.756299 mapwrite 0x8cfbdd thru 0x8cfeae (0x2d2 bytes) All operations completed A-OK! PASS 16b (24s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:45:32 (1773668732) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:45:35 (1773668735) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.351295 s, 23.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.302756 s, 27.7 MB/s ldlm.namespaces.MGC192.168.201.152@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff95bd46192800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff95bd4a140800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff95bd46192800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff95bd46192800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.98889 s, 13.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.56316 s, 40.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.10154 s, 49.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.81662 s, 21.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20603 s, 20.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.70428 s, 38.8 MB/s PASS 16d (35s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:46:10 (1773668770) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.208801 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885531 s, 11.8 MB/s PASS 16e (5s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:46:15 (1773668775) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:46:41 (1773668801) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 09:47:07 (1773668827) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 09:47:13 (1773668833) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 09:47:19 (1773668839) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51595 s, 17.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.821725 s, 31.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.9092 s, 13.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.01191 s, 25.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27989 s, 20.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27597 s, 20.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57376 s, 16.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.989666 s, 26.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.1874 s, 22.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.76113 s, 14.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.89021 s, 13.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.28314 s, 20.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.38877 s, 18.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.63329 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.71301 s, 15.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.43492 s, 18.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55337 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.92265 s, 13.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0422 s, 12.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (25s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:47:44 (1773668864) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Mar 16 09:48:17 EDT 2026 + fsx 2 done: rc=0 Mon Mar 16 09:48:17 EDT 2026 + fsx 3 done: rc=0 Mon Mar 16 09:48:17 EDT 2026 + fsx 4 done: rc=0 Mon Mar 16 09:48:17 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Mon Mar 16 09:48:17 EDT 2026 PASS 16k (42s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:48:26 (1773668906) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 09:48:36 (1773668916) mmap test1: basic mmap operation (PASS, 0.11338s) mmap test2: MAP_PRIVATE not write back (PASS, 0.092727s) mmap test3: concurrent mmap ops on two nodes (PASS, 12.113s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.531s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1579s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22061s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1537s) mmap test11: multi-thread shared mmap (PASS, 1.167s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1029s) PASS 18 (27s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:49:03 (1773668943) oleg152-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:49:07 (1773668947) PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:49:12 (1773668952) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 09:49:18 (1773668958) now is 1773668959 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 new atime is 1773669020 PASS 23 (66s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:50:24 (1773669024) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3760128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 9216 3760128 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 9.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 40.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117968 400 117568 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3760128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 228.9K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117968 400 117568 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117968 400 117568 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234818 386 234432 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 9216 3760128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 9216 3760128 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 PASS 24a (5s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:50:29 (1773669029) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:50:32 (1773669032) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (4s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:50:37 (1773669037) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 09:50:38 (1773669038) utime: good mknod times 1773669038 <= 1773669039 <= 1773669039 for /mnt/lustre/f26a utime: good mknod times 1773669038 <= 1773669039 <= 1773669039 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (5s) == sanityn test 26b: sync mtime between ost and mds ====== 09:50:43 (1773669043) PASS 26b (6s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:50:49 (1773669049) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-16 09:50:49.000000000 -0400 Birth: 2026-03-16 09:50:49.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-16 09:50:49.000000000 -0400 Birth: 2026-03-16 09:50:49.000000000 -0400 PASS 26c (5s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:50:55 (1773669055) 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:51:01 (1773669061) fail_loc=0x314 PASS 31a (5s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:51:06 (1773669066) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg152-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.152@tcp:/lustre 7542784 41984 7496704 1% /mnt/lustre2 PASS 31b (16s) == sanityn test 31r: open-rename(replace) race =========== 09:51:22 (1773669082) -rw-r--r-- 1 root root 0 Mar 16 09:51 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:51:28 (1773669088) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42886 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:51:34 (1773669094) PASS 31t (5s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:51:41 (1773669101) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:51:43 (1773669103) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:51:45 (1773669105) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:51:48 (1773669108) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 09:51:50 (1773669110) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid in IDLE state after 0 sec oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid in FULL state after 0 sec oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid in IDLE state after 0 sec oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid in FULL state after 0 sec oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95bd46192800.ost_server_uuid in IDLE state after 0 sec oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95bd46192800.ost_server_uuid in FULL state after 0 sec PASS 34 (152s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:54:23 (1773669263) fail_loc=0x80000317 Race attempt 0 Wait for 48944 48959 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1746: 48944 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.152@tcp:/lustre 7.2G 42M 7.2G 1% /mnt/lustre 192.168.201.152@tcp:/lustre 7.2G 42M 7.2G 1% /mnt/lustre2 - open/close 581 (time 1773669282.11 total 10.01 last 58.06) - open/close 1180 (time 1773669292.11 total 20.01 last 59.90) - open/close 1859 (time 1773669302.12 total 30.01 last 67.85) - open/close 2476 (time 1773669312.12 total 40.02 last 61.67) - open/close 3056 (time 1773669322.13 total 50.02 last 57.98) - open/close 3758 (time 1773669332.14 total 60.03 last 70.14) total: 4000 open/close in 65.73 seconds: 60.85 ops/second PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:55:39 (1773669339) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg152-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01135 s, 51.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7495680) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.674776 s, 77.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7486464) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.695999 s, 75.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7430144) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29171 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7428096) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.79327 s, 66.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.6318 s, 83.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7486464) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.740138 s, 70.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7489536) after_dd(7418880) after(7470080) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.98041 s, 53.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7470080) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.722657 s, 72.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7486464) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.824445 s, 63.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7488512) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.782977 s, 67.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7489536) after_dd(7437312) after(7488512) PASS 36 (309s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 10:00:48 (1773669648) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 - create 1259 (time 1773669663.23 total 10.00 last 125.87) - create 2647 (time 1773669673.24 total 20.01 last 138.72) - create 3857 (time 1773669683.24 total 30.01 last 120.98) - create 5075 (time 1773669693.24 total 40.01 last 121.80) - create 6518 (time 1773669703.25 total 50.02 last 144.18) - create 8008 (time 1773669713.26 total 60.03 last 148.85) - create 9425 (time 1773669723.27 total 70.04 last 141.54) total: 10000 create in 74.22 seconds: 134.74 ops/second PASS 37 (86s) == sanityn test 39a: file mtime does not change after rename ========================================================== 10:02:14 (1773669734) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 10:02:19 (1773669739) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 10:02:26 (1773669746) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 10:02:33 (1773669753) fail_loc=0x411 fail_loc=0 PASS 39d (5s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 10:02:40 (1773669760) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (15s) == sanityn test 40c: pdirops: link and others ======================================================================== 10:02:55 (1773669775) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 10:03:10 (1773669790) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 10:03:25 (1773669805) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (13s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 10:03:38 (1773669818) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 10:03:49 (1773669829) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 10:04:01 (1773669841) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 10:04:13 (1773669853) Conflict PASS 41d (9s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 10:04:22 (1773669862) Conflict PASS 41e (11s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 10:04:33 (1773669873) Conflict PASS 41f (9s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 10:04:42 (1773669882) Conflict PASS 41g (10s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 10:04:52 (1773669892) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (9s) == sanityn test 41i: reint_open: create vs create ======== 10:05:01 (1773669901) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (775s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:17:57 (1773670677) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:18:03 (1773670683) open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:18:10 (1773670690) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:18:17 (1773670697) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:18:24 (1773670704) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:18:30 (1773670710) Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:18:37 (1773670717) Conflict PASS 42g (7s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:18:44 (1773670724) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:18:51 (1773670731) PASS 43a (31s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:19:22 (1773670762) Conflict PASS 43b (7s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:19:29 (1773670769) Conflict PASS 43c (8s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:19:37 (1773670777) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (7s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:19:44 (1773670784) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:19:51 (1773670791) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (6s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:19:57 (1773670797) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:20:04 (1773670804) Conflict PASS 43h (7s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:20:11 (1773670811) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:20:12 (1773670812) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (60s) == sanityn test 43k: unlink vs create ==================== 10:21:12 (1773670872) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (541s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:30:13 (1773671413) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Mar 16 10:30:17 EDT 2026 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:30:20 (1773671420) open: File exists Conflict PASS 44b (9s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:30:29 (1773671429) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (11s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:30:40 (1773671440) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:30:51 (1773671451) Conflict PASS 44e (11s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:31:02 (1773671462) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:31:11 (1773671471) Conflict PASS 44g (8s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:31:19 (1773671479) Conflict PASS 44h (8s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:31:28 (1773671488) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:31:29 (1773671489) PASS 45a (68s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:32:37 (1773671557) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:32:44 (1773671564) Conflict PASS 45c (7s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:32:51 (1773671571) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:32:58 (1773671578) Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:33:06 (1773671586) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:33:13 (1773671593) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:33:21 (1773671601) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:33:27 (1773671607) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 10:33:28 (1773671608) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (787s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 10:46:35 (1773672395) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (20s) == sanityn test 46b: pdirops: link vs create ======================================================================== 10:46:55 (1773672415) open: File exists Conflict PASS 46b (19s) == sanityn test 46c: pdirops: link vs link ======================================================================== 10:47:14 (1773672434) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (21s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 10:47:35 (1773672455) Conflict PASS 46d (21s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 10:47:56 (1773672476) Conflict PASS 46e (21s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 10:48:17 (1773672497) Conflict PASS 46f (19s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 10:48:36 (1773672516) Conflict PASS 46g (20s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 10:48:56 (1773672536) Conflict PASS 46h (22s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 10:49:18 (1773672558) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 10:49:24 (1773672564) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (6s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 10:49:30 (1773672570) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 10:49:37 (1773672577) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 10:49:43 (1773672583) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 10:49:48 (1773672588) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 10:49:54 (1773672594) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (9s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 10:50:02 (1773672602) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (6s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 10:50:08 (1773672608) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.26911 s, 38.1 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (18s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 10:50:26 (1773672626) 0+1 records in 0+1 records out 158 bytes copied, 0.0266779 s, 5.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216254) - No such process PASS 51a (12s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 10:50:38 (1773672638) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0610465 s, 16.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290812 s, 3.6 MB/s 3145728 PASS 51b (26s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:51:04 (1773672664) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0591441 s, 17.3 kB/s PASS 51c (18s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:51:22 (1773672682) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117588 s, 8.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (13s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:51:36 (1773672696) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (14s) == sanityn test 54: rename locking ======================= 10:51:50 (1773672710) llite.lustre-ffff95bd46192800.xattr_cache=1 llite.lustre-ffff95bd4a140800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:51:53.000000000 -0400 Modify: 2026-03-16 10:51:53.000000000 -0400 Change: 2026-03-16 10:51:53.000000000 -0400 Birth: 2026-03-16 10:51:53.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:52:00.000000000 -0400 Modify: 2026-03-16 10:52:02.000000000 -0400 Change: 2026-03-16 10:52:02.000000000 -0400 Birth: 2026-03-16 10:52:00.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:52:08.000000000 -0400 Modify: 2026-03-16 10:52:08.000000000 -0400 Change: 2026-03-16 10:52:08.000000000 -0400 Birth: 2026-03-16 10:52:07.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:52:07.000000000 -0400 Modify: 2026-03-16 10:52:09.000000000 -0400 Change: 2026-03-16 10:52:09.000000000 -0400 Birth: 2026-03-16 10:52:07.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:52:15.000000000 -0400 Modify: 2026-03-16 10:52:17.000000000 -0400 Change: 2026-03-16 10:52:17.000000000 -0400 Birth: 2026-03-16 10:52:15.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 10:52:15.000000000 -0400 Modify: 2026-03-16 10:52:17.000000000 -0400 Change: 2026-03-16 10:52:17.000000000 -0400 Birth: 2026-03-16 10:52:15.000000000 -0400 PASS 54 (41s) == sanityn test 55a: rename vs unlink target dir ========= 10:52:31 (1773672751) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (18s) == sanityn test 55b: rename vs unlink source dir ========= 10:52:49 (1773672769) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (17s) == sanityn test 55c: rename vs unlink orphan target dir == 10:53:06 (1773672786) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (24s) == sanityn test 55d: rename file vs link ================= 10:53:31 (1773672811) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (21s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:53:52 (1773672832) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (6s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:53:58 (1773672838) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (28s) == sanityn test 55g: rename: race with trylock =========== 10:54:27 (1773672867) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (32s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:54:59 (1773672899) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1773672902 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.379125 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (175s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 10:57:53 (1773673073) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1773673085 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.748922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (189s) == sanityn test 60: Verify data_version behaviour ======== 11:01:02 (1773673262) PASS 60 (19s) == sanityn test 70a: cd directory && rm directory ======== 11:01:21 (1773673281) PASS 70a (11s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 11:01:33 (1773673293) PASS 70b (12s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 11:01:45 (1773673305) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0173398 s, 3.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (6s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 11:01:51 (1773673311) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0231643 s, 2.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (6s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 11:01:57 (1773673317) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (6s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 11:02:03 (1773673323) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0851366 s, 770 kB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 11:02:08 (1773673328) llite.lustre-ffff95bd46192800.xattr_cache=1 llite.lustre-ffff95bd4a140800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (12s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 11:02:20 (1773673340) llite.lustre-ffff95bd46192800.xattr_cache=1 llite.lustre-ffff95bd4a140800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (10s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 11:02:31 (1773673351) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0386133 s, 26.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0125815 s, 81.4 kB/s 229111: get lock2 229115: get lock1 229111: done 229111: get lock1 229115: done 229115: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229115: cannot get lock2: Resource deadlock avoided 229115: exit rc=0 229111: done 229111: exit rc=0 PASS 74 (15s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 11:02:45 (1773673365) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0422811 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214061 s, 49.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229696 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (15s) == sanityn test 76: Verify MDT open_files listing ======== 11:03:00 (1773673380) opening files: ........ closing files: ........ PASS 76 (94s) == sanityn test 77a: check FIFO NRS policy =============== 11:04:34 (1773673474) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3152 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617502 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283614 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109214 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531749 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549462 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688505 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261013 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538865 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807778 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196522 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163718 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237306 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15687 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27868 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297434 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376761 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133058 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873573 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964243 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142286 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152258 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140012 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158032 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15776 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909901 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134185 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141219 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15227 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223037 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177624 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12382 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137464 s, 7.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (13s) == sanityn test 77b: check CRR-N NRS policy ============== 11:04:47 (1773673487) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.447022 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553662 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393405 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10986 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456095 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188163 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148968 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411559 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471458 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506611 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107987 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952399 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211862 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149667 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189666 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664484 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941695 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110984 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702176 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134092 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014401 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134192 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525574 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461423 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147128 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730331 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100642 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951606 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138786 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156044 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907542 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140887 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100927 s, 104 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428231 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718398 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670785 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512155 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2363 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144974 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154012 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157441 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267979 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250324 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298125 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450032 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374188 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501305 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488348 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401926 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424353 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571729 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401726 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184429 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177649 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017868 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143478 s, 73.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2412 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135633 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132289 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225714 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926782 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31464 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102059 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927735 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081422 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758284 s, 13.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (20s) == sanityn test 77c: check ORR NRS policy ================ 11:05:07 (1773673507) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.472975 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687245 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697953 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100469 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518539 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101965 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13476 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871874 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683578 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212865 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307319 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321342 s, 3.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265127 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150154 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133268 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40129 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382813 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499689 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527366 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151675 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00866121 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872716 s, 120 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113999 s, 92.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103578 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201534 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00764226 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184546 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715901 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21662 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137902 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164797 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185559 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392715 s, 26.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.398144 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481733 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680378 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119734 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157122 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20065 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118843 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146263 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180478 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806071 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241521 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185324 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289712 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291997 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324542 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166209 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368029 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147866 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177603 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152524 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224574 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20337 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125146 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178705 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127816 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160793 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154468 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250826 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246768 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152044 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155616 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191498 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216803 s, 4.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.534692 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656528 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541009 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118607 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111544 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867652 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110229 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155182 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181811 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278126 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288629 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140227 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300114 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215506 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228603 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160317 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22202 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534388 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0026844 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727817 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924477 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158294 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135545 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091857 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109307 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285621 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931433 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121302 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354622 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17984 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767876 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551819 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835948 s, 12.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (23s) == sanityn test 77d: check TRR nrs policy ================ 11:05:30 (1773673530) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430847 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469153 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502367 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821142 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537047 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611201 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874702 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066113 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298157 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15842 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905592 s, 11.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170956 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183908 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149418 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105599 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165838 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221016 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477312 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453427 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143347 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714639 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601935 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635127 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747073 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804999 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712164 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793255 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102578 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166915 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124407 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183763 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114535 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15472 s, 6.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.528063 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191516 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209689 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220473 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29555 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257861 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330194 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324881 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402176 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203017 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380324 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419333 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560177 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363602 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45752 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334863 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.721286 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139799 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148586 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191261 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159969 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175493 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196087 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169427 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192882 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217929 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194883 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206335 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218087 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192201 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226741 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318387 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223367 s, 4.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.735105 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238889 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537733 s, 19.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330204 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107822 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176928 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259221 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231701 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352058 s, 3.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232963 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281362 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343893 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346378 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374678 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370765 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364619 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419807 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982924 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360225 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998654 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130214 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122432 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135245 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148504 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137613 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965926 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174134 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158453 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141428 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16928 s, 6.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149776 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166672 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188496 s, 5.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (25s) == sanityn test 77e: check TBF NID nrs policy ============ 11:05:56 (1773673556) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.52@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.433691 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501065 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103763 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096099 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157887 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012112 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039961 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559923 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357861 s, 29.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192325 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268513 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292099 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3241 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347567 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262015 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212496 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400568 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00320816 s, 327 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019865 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00316381 s, 331 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033294 s, 315 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156876 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00283434 s, 370 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020537 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150928 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560563 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437913 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.554694 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196279 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372586 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182595 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17235 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136751 s, 7.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619561 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529004 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639061 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11256 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640084 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643956 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124324 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187938 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129639 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166613 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043558 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16693 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767551 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317998 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172493 s, 6.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104886 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206585 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690281 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694262 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237196 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626114 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835627 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084681 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884245 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115731 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171356 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607097 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866114 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969755 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115994 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125238 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118927 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103227 s, 10.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.413576 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577301 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463565 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132203 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131109 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138097 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958525 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239861 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204349 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201637 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365444 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358657 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282439 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278941 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299342 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261313 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236787 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943325 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785744 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011808 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950183 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616121 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065104 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106283 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077713 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173454 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958603 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816154 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167085 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100085 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383301 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132673 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165179 s, 6.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493008 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862083 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952126 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114412 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999097 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148157 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157124 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892606 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148824 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225091 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137033 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204666 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260581 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215124 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26564 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313633 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316373 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320318 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104195 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692085 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408046 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052926 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913803 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060209 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669696 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109609 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780387 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817449 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927031 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170813 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137079 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102902 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12653 s, 8.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (38s) == sanityn test 77f: check TBF JobID nrs policy ========== 11:06:35 (1773673595) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.693059 s, 24.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314107 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990846 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104775 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375407 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129425 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180196 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142777 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255232 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256701 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357613 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225389 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427287 s, 2.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236898 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26378 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4303 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372617 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256635 s, 409 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11252 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00877624 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00264667 s, 396 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116253 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204831 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215151 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196434 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165667 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131773 s, 79.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169282 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184598 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730151 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103928 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042603 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575674 s, 18.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434514 s, 38.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368647 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707198 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100217 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103384 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632364 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121622 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663495 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267649 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682951 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16395 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162604 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295989 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119169 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190428 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177897 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209159 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137604 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502858 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705738 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916311 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767135 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07987 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148198 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708867 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760963 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972145 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107706 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639183 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120638 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682777 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156899 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297509 s, 35.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.429817 s, 39.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480758 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593351 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11461 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180247 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319261 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268504 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255524 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347257 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325222 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331544 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303691 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251179 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342741 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339083 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452209 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458727 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182122 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124485 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115773 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17414 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263632 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276578 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208095 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211129 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165776 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274312 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341929 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141138 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10282 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896517 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678416 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503176 s, 20.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571775 s, 29.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895506 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131534 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227848 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207421 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150333 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775307 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212857 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343238 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381669 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322368 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327977 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29708 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153605 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164356 s, 6.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33874 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255852 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121902 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385782 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994975 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100351 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156076 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112664 s, 93.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993419 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305871 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128267 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877984 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947756 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163538 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156129 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157728 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216573 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107393 s, 9.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (38s) == sanityn test 77g: Change TBF type directly ============ 11:07:12 (1773673632) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (16s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 11:07:28 (1773673648) oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg152-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg152-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg152-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg152-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 PASS 77h (19s) == sanityn test 77i: Change rank of TBF rule ============= 11:07:48 (1773673668) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (34s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 11:08:22 (1773673702) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5271 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767401 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566074 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114592 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377286 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843632 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10563 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143172 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545329 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095339 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210122 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192632 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234999 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145859 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203824 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248426 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22531 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12109 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107761 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107301 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121121 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785102 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112617 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816568 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111966 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916799 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137189 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141941 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931897 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168714 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162824 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105999 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126842 s, 8.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6761 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7193 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (74s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 11:09:36 (1773673776) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33207 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078547 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713458 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672424 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838769 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124738 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124669 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187458 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19094 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184205 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163954 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180417 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268159 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230088 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286018 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316822 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355915 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668317 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125693 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10937 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138589 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958349 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11926 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116441 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140603 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546622 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113712 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129721 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810044 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854892 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121899 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125849 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122686 s, 8.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.9256 s, 5.0 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0217 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29261 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111799 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665376 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178538 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1944 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157073 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265001 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370324 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429796 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285465 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386439 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326139 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433929 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486363 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310482 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365658 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135946 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144369 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144872 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1539 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16139 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172356 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17074 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156119 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163889 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180898 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188507 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171944 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177034 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202411 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185409 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210833 s, 5.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1935 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0052 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (163s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 11:12:19 (1773673939) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2864 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103826 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121939 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176621 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251021 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239211 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856834 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551821 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233447 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129378 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291317 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646944 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324383 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300745 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28462 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501827 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290976 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792348 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106068 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609938 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835612 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470871 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836927 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130321 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114642 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148028 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865801 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101481 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960032 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125902 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112982 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138669 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238154 s, 44.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1156 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9897 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30693 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540627 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146252 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163187 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164174 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179353 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208205 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179819 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218738 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23717 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22784 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245952 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142991 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137533 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372132 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399719 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430448 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166985 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148209 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120419 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192233 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140201 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199019 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197274 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183816 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219328 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190827 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213108 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181003 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257214 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235264 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21666 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247847 s, 4.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4458 s, 5.1 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8845 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (164s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 11:15:04 (1773674104) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741876 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596324 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478263 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762293 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12872 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685754 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103321 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122165 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273569 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101066 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953491 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164845 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169396 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386514 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274739 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227125 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336464 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161526 s, 64.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248855 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141574 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362399 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426905 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524585 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548083 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604366 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709113 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759688 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11859 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881469 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890471 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107457 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1084 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142089 s, 7.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7465 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97018 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.52@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.52@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.707557 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986474 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086391 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13815 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173057 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174885 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235862 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212717 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234198 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256327 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249878 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304551 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240114 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239609 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31319 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283604 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348582 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110249 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183273 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845081 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136712 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296987 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764132 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445127 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901829 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704104 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0064911 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181426 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152962 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112406 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157627 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173858 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022594 s, 46.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.617 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.041 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.52@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479719 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468404 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336565 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09839 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532415 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124019 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16215 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21418 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15715 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214031 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191907 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290377 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321835 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235999 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279165 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353038 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412161 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144468 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149575 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659109 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115822 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994014 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797021 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197957 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114205 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108313 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187654 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189457 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14634 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144584 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163071 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229465 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228084 s, 4.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3803 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43641 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642084 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454652 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557398 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698302 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401007 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14797 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193151 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173182 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198102 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232746 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306804 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253333 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351819 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239291 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296778 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410686 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366041 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533178 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504732 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123332 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615032 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524192 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475147 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744504 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069391 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854192 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777304 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106544 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090951 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939678 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988897 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101938 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104943 s, 10.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9302 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97569 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.095 s, 6.9 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0125 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08914 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044583 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117943 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153229 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772924 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972932 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125057 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757954 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942838 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214416 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22678 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246527 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311522 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232338 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188595 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358855 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396551 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736252 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864129 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651667 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119889 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643134 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833191 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717537 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807974 s, 13.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105621 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923633 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898888 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895592 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098528 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984235 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100914 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117483 s, 8.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.7921 s, 5.0 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.658 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741079 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275331 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784135 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110431 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126307 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130172 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227176 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220718 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240343 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245902 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289348 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195094 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23587 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191623 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201579 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221279 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349812 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438759 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500882 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560285 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161887 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585819 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621386 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115376 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745304 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091651 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137585 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102938 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110245 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111703 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11963 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140134 s, 7.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5044 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0067 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.826542 s, 20.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699895 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604264 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07847 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195781 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907371 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149794 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215541 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212823 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274773 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295624 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24463 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303517 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2918 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247338 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338426 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312638 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374085 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983035 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104657 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704969 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778857 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695298 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619169 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870782 s, 12.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881418 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835301 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896472 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101529 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104026 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171839 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110268 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216433 s, 4.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7992 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98217 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.8618 s, 7.6 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0921 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (487s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 11:23:11 (1773674591) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.529093 s, 31.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104938 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113371 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140071 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169456 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242248 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264661 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126356 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202834 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176916 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312402 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211994 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356467 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317597 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264319 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392113 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29539 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106175 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038559 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103604 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450214 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117621 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562923 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191202 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118242 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152901 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112683 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764031 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878291 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949479 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939621 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109895 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125118 s, 8.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8362 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0271 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (59s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 11:24:10 (1773674650) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.52@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.52@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57587 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749574 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141004 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19225 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244152 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25772 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269438 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240693 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20157 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160434 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195407 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306932 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278102 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269679 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311914 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249084 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279072 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830181 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916321 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709817 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071474 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457657 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111885 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144437 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690428 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109118 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230575 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913751 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111849 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210483 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670175 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107656 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116955 s, 9.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9428 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9324 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 11:25:05 (1773674705) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.52@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815748 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435486 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843102 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107966 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453407 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795898 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110129 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211679 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262029 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069159 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182331 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289268 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295438 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345689 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173258 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19324 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344052 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928688 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602614 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691864 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865114 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017009 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738453 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119845 s, 87.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420722 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433135 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0077359 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791341 s, 133 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00796833 s, 132 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046532 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010112 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00549984 s, 191 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129368 s, 81.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7559 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38729 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (52s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 11:25:57 (1773674757) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.608053 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767928 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108022 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493391 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148903 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986495 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170594 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208274 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182888 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217097 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261054 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182262 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209968 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324879 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389228 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38824 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443734 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636032 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103327 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662564 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610535 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692056 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557057 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713098 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712583 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657914 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849519 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583482 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106725 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856843 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097371 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834217 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121273 s, 8.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.5536 s, 7.2 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.037 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.57432 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99361 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (105s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 11:27:42 (1773674862) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15429 s, 14.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122712 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123191 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335117 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217079 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284749 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423559 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367244 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29878 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47709 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352532 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334357 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48214 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428127 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.557783 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386461 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518091 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140908 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264193 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159381 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15915 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141868 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302841 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213603 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199832 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23583 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266133 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387672 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272736 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270661 s, 3.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285281 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276229 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346138 s, 3.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.1148 s, 5.0 MB/s Write runtime is 23 s, speed is 4.347826 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0368 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (85s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 11:29:08 (1773674948) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.708661 s, 23.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051063 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670483 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150766 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143774 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160135 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172969 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248531 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254287 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208834 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155954 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181809 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208309 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200797 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179319 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960047 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff95bd4a140800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233058 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291922 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106049 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279572 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014264 s, 73.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078846 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810709 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816775 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801753 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00267174 s, 392 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd4a140800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846072 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372438 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948239 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279748 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155993 s, 67.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044257 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184902 s, 56.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5804 s, 8.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0101 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg152-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.790674 s, 21.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140346 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118287 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177193 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232563 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30157 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276532 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282705 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345236 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168305 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255504 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230097 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371405 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277018 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312062 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339547 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42021 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330864 s, 31.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347809 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110523 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407343 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853278 s, 12.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102982 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093876 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060104 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104426 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126264 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787463 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832596 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826352 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890363 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110002 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127664 s, 8.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2013 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99684 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.3943 s, 7.8 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0136 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (160s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 11:31:48 (1773675108) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre opts: Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg152-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.52,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg152-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.52,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg152-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.52,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/61262/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry2 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.52,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg152-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Started clients oleg152-client.virtnet: 192.168.201.152@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Started clients oleg152-client.virtnet: 192.168.201.152@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95bd4993b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff95bd505c5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd4993b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd505c5000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.899534 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11557 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143141 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20646 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114323 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270226 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135212 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202569 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240573 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287769 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277537 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361279 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45152 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41979 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446072 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404233 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35997 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd505c5000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393226 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395485 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446881 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401011 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392785 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4777 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408817 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462352 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462928 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421374 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4201 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42653 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420127 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43325 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490786 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502264 s, 2.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.739 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.72071 s, 18.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.658361 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095582 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125312 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131715 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111772 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595238 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903461 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215359 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177113 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542045 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167275 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24287 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361062 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336704 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336368 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412607 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438799 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd505c5000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179272 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167311 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184045 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219268 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199053 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335473 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212277 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328025 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242914 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256101 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259502 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353926 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244234 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271118 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236716 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238647 s, 4.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg152-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg152-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4872 s, 8.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.066 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (287s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 11:36:36 (1773675396) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg152-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (25s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 11:37:01 (1773675421) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (16s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 11:37:17 (1773675437) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg152-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.80143 s, 276 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg152-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0829 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (60s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 11:38:17 (1773675497) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.787477 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13746 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188272 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142837 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116798 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163495 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157925 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303264 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485975 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215778 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445777 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525405 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325286 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305256 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498591 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.550471 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325741 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd505c5000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff95bd505c5000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113782 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577482 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101917 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559593 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107572 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652913 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568558 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533519 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663161 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137618 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740113 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137037 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725039 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804271 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838918 s, 12.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3792 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41101 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.847922 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811088 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103531 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132638 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135959 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233753 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183426 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2714 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290778 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289208 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276576 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd505c5000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff95bd505c5000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360271 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375859 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309748 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500649 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449641 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.621987 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff95bd505c5000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231569 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275984 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333273 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239453 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225949 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196927 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20848 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289935 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219673 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268663 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296219 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274448 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267478 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460963 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236231 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268141 s, 3.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg152-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg152-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9812 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52681 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (96s) == sanityn test 77o: Changing rank should not panic ====== 11:39:53 (1773675593) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (18s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 11:40:12 (1773675612) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (149s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 11:42:41 (1773675761) mds.MDS.mdt.nrs_policies=tbf oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg152-client: oleg152-server: ssh exited with exit code 36 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg152-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (48s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 11:43:29 (1773675809) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 266044 kill md_thread xargs PASS 77r (59s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 11:44:29 (1773675869) oleg152-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg152-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg152-client: oleg152-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 11:44:39 (1773675879) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826934630 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 11:44:42.000000000 -0400 Modify: 2026-03-16 11:44:42.000000000 -0400 Change: 2026-03-16 11:44:42.000000000 -0400 Birth: 2026-03-16 11:44:41.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 11:44:58 (1773675898) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 11:45:01 (1773675901) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 11:45:04 (1773675904) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 11:45:08 (1773675908) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 11:45:11 (1773675911) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 11:45:16 (1773675916) - open/close 666 (time 1773675931.63 total 10.02 last 66.48) - open/close 1297 (time 1773675941.63 total 20.02 last 63.07) - open/close 1967 (time 1773675951.65 total 30.04 last 66.92) total: 2500 open/close in 38.22 seconds: 65.42 ops/second snapshot_time 1773676062.026159148 secs.nsecs start_time 1773675918.810939104 secs.nsecs elapsed_time 143.215220044 secs.nsecs open 2505 samples [usecs] 148 15499 3870792 10409006174 close 2504 samples [usecs] 174 8042 1133048 1074677578 mknod 2501 samples [usecs] 459 15259 3326462 7989313444 unlink 2500 samples [usecs] 564 105044 4750839 31435769977 mkdir 2 samples [usecs] 1334 2781 4115 9513517 rmdir 2 samples [usecs] 2486 43858 46344 1929704360 rename 2500 samples [usecs] 461 58091 18022387 265994983705 getattr 3761 samples [usecs] 8 12745 305413 511220981 parallel_rename_file 2500 samples [usecs] 461 58091 18022387 265994983705 crossdir_rename 2500 samples [usecs] 461 58091 18022387 265994983705 PASS 81d (152s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 11:47:48 (1773676068) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7572 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7572 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 11:47:56 (1773676076) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 11:48:01 (1773676081) fail_loc=0x8000060b PASS 84 (15s) == sanityn test 85: Lustre API root cache race =========== 11:48:16 (1773676096) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1773676098 Finishing test test1 at 1773676103 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 11:48:29 (1773676109) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 11:48:32 (1773676112) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 11:48:36 (1773676116) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 11:48:39 (1773676119) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5270 0x1496 0x240000400 1 5232 0x1470 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5233 0x1471 0x280000400 0 5271 0x1497 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 11:48:56 (1773676136) osc.lustre-OST0000-osc-ffff95bd4993b000.idle_timeout=0 osc.lustre-OST0000-osc-ffff95bd505c5000.idle_timeout=0 osc.lustre-OST0001-osc-ffff95bd4993b000.idle_timeout=0 osc.lustre-OST0001-osc-ffff95bd505c5000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.263665 s, 31.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.02471 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 280918 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.071614 s, 57.2 kB/s osc.lustre-OST0000-osc-ffff95bd4993b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff95bd505c5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd4993b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95bd505c5000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 11:49:12 (1773676152) llite.lustre-ffff95bd4993b000.fast_read=0 llite.lustre-ffff95bd505c5000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0840995 s, 24.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff95bd4993b000.fast_read=1 llite.lustre-ffff95bd505c5000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 11:49:32 (1773676172) llite.lustre-ffff95bd4993b000.fast_read=0 llite.lustre-ffff95bd505c5000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0168309 s, 730 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff95bd4993b000.fast_read=1 llite.lustre-ffff95bd505c5000.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 11:49:47 (1773676187) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 11:49:51 (1773676191) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00803901 s, 510 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 11:49:58 (1773676198) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2497 s, 8.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 11:50:07 (1773676207) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227457 s, 9.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 11:50:15 (1773676215) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00659043 s, 1.8 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 11:50:23 (1773676223) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0220406 s, 186 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 11:50:31 (1773676231) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0136034 s, 301 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 11:50:38 (1773676238) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0218714 s, 187 kB/s PASS 101c (9s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 11:50:49 (1773676249) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1773676252 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1773676256 Finishing test test23 at 1773676256 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 11:51:03 (1773676263) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 11:51:07 (1773676267) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 11:51:36 (1773676296) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 11:51:40 (1773676300) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0348206 s, 29.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937165 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 11:51:42.000000000 -0400 Modify: 2026-03-16 11:51:43.000000000 -0400 Change: 2026-03-16 11:51:43.000000000 -0400 Birth: 2026-03-16 11:51:42.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826937165 -rw-r--r-- 1 0 0 1773676302 1773676302 1773676303 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937165 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 11:51:42.000000000 -0400 Modify: 2026-03-16 11:51:43.000000000 -0400 Change: 2026-03-16 11:51:43.000000000 -0400 Birth: 2026-03-16 11:51:42.000000000 -0400 1773676303 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 11:51:49 (1773676309) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 11:51:57 (1773676317) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.389399 s, 26.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5279 0x149f 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 11:52:07 (1773676327) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.328075 s, 32.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5240 0x1478 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7572 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 11:52:21 (1773676341) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0150184 s, 666 kB/s llite.lustre-ffff95bd4993b000.hybrid_io=0 llite.lustre-ffff95bd505c5000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.65943 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 11:52:34 (1773676354) 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 192.168.201.152@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre2 PASS 109 (1072s) == sanityn test 110: do not grant another lock on resend ========================================================== 12:10:26 (1773677426) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 12:10:27.000000000 -0400 Modify: 2026-03-16 12:10:27.000000000 -0400 Change: 2026-03-16 12:10:27.000000000 -0400 Birth: 2026-03-16 12:10:27.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg152-client.virtnet oleg152-client.virtnet PASS 110 (67s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 12:11:33 (1773677493) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 12:11:36 (1773677496) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 12:11:38 (1773677498) lustre-OST0000-osc-ffff95bd4609e800 active. lustre-OST0001-osc-ffff95bd4609e800 active. lustre-OST0000-osc-ffff95bd4609e800 active. lustre-OST0001-osc-ffff95bd4609e800 active. MGC192.168.201.152@tcp active. PASS 113 (4s) == sanityn test 114: implicit default LMV inherit ======== 12:11:42 (1773677502) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 12:11:44 (1773677504) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 12:11:46 (1773677506) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 12:11:48 (1773677508) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00761074 s, 2.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0449472 s, 456 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (6s) == sanityn test 200: service remains healthy while able to process request ========================================================== 12:11:53 (1773677513) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0698 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1183 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.169 s, 0.2 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1318 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1663 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1564 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1456 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1055 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1753 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.173 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1668 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2068 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2091 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.201 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2494 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3079 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2996 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3093 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95bd43910800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95bd43910800.ost_server_uuid in IDLE state after 0 sec PASS 200 (71s) cleanup: ====================================================== == sanityn test complete, duration 9280 sec ============== 12:13:06 (1773677586) === sanityn: start cleanup 12:13:07 (1773677587) === Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg152-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 12:15:19 (1773677719) ===