-----============= acceptance-small: sanityn ============----- Thu Jan 22 04:13:25 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg345-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 04:13:39 (1769073219) === Starting client oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Mount client oleg345-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Started clients oleg345-client.virtnet: 192.168.203.145@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg345-client.virtnet: executing check_config_client /mnt/lustre oleg345-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg345-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bd6c9830000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8bd6d5282800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6c9830000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6d5282800.idle_timeout=debug disable quota as required oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 04:14:00 (1769073240) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230318 s, 45.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7531] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 04:14:02 (1769073242) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 04:14:07 (1769073247) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (4s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 04:14:11 (1769073251) -rw-r--r-- 1 root root 0 Jan 22 04:14 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 04:14:15 (1769073255) -rw-r--r-- 1 root root 0 Jan 22 04:14 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 04:14:19 (1769073259) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 04:14:22 (1769073262) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 04:14:26 (1769073266) total 1 -rwxrwxrwx 1 root root 0 Jan 22 04:14 f2a -rwxrwxrwx 1 root root 0 Jan 22 04:14 f2b total 1 -rwxrwxrwx 1 root root 0 Jan 22 04:14 f2a -rwxrwxrwx 1 root root 0 Jan 22 04:14 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00763275 s, 67.1 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 04:14:30 (1769073270) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 04:14:33 (1769073273) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.214183 s, 9.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 04:14:37 (1769073277) PASS 3 (4s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 04:14:41 (1769073281) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 04:14:45 (1769073285) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (4s) == sanityn test 6: remove of open file on other node ============================================================================ 04:14:49 (1769073289) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 04:14:53 (1769073293) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 04:14:56 (1769073296) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 04:15:00 (1769073300) PASS 9a (10s) == sanityn test 9b: append to striped sparse file ======== 04:15:10 (1769073310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102911 s, 10.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0381534 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (13s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 04:15:23 (1769073323) 1+0 records in 1+0 records out 1 byte copied, 0.00745417 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00883343 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00961853 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00863046 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00748218 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0077625 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0230768 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0731572 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.11502 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119307 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0085598 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017131 s, 0.1 kB/s PASS 10a (11s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 04:15:34 (1769073334) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0138114 s, 222 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0534077 s, 76.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000438692 s, 7.0 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 04:15:43 (1769073343) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (10s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 04:15:53 (1769073353) warning: '-runtime' deprecated, use '-t runtime' instead - link 231 (time 1769073365.81 total 10.02 last 23.04) - link 454 (time 1769073375.86 total 20.08 last 22.18) - link 737 (time 1769073385.86 total 30.08 last 28.29) - link 1002 (time 1769073395.88 total 40.09 last 26.46) - link 1327 (time 1769073405.88 total 50.10 last 32.48) - link 1619 (time 1769073415.94 total 60.15 last 29.04) - link 1952 (time 1769073425.97 total 70.18 last 33.20) - link 2290 (time 1769073435.97 total 80.18 last 33.80) - link 2578 (time 1769073445.98 total 90.20 last 28.75) total: 2916 link in 100.00 seconds: 29.16 ops/second using seed 3077273620 running for 100 seconds - stat 3414 (time 1769073367 ; total 11 ; last 11) - stat 8092 (time 1769073378 ; total 22 ; last 11) - stat 10000 (time 1769073381 ; total 25 ; last 3) - stat 14620 (time 1769073392 ; total 36 ; last 11) - stat 20000 (time 1769073402 ; total 46 ; last 10) - stat 25103 (time 1769073413 ; total 57 ; last 11) - stat 27969 (time 1769073424 ; total 68 ; last 11) - stat 30000 (time 1769073430 ; total 74 ; last 6) - stat 33076 (time 1769073441 ; total 85 ; last 11) - stat 36770 (time 1769073452 ; total 96 ; last 11) total: 38090 stats in 100 seconds: 380.899994 stats/second - unlinked 0 (time 1769073406 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2916) error: No such file or directory total: 2915 unlinks in 82 seconds: 35.548782 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21321) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21323) - No such process PASS 12 (142s) == sanityn test 13: test directory page revocation ======= 04:18:15 (1769073495) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 04:18:24 (1769073504) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 22509 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 04:18:32 (1769073512) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 23088 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 04:18:39 (1769073519) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23666 Terminated $DIR1/$tdir/sleep 60 PASS 14b (9s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 04:18:48 (1769073528) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24249 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 04:18:57 (1769073537) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24831 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 04:19:05 (1769073545) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 04:19:56 (1769073596) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.07575 s, 7.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.680053 s, 12.3 MB/s Chance of close/open is 1 in 50 Seed set to 3602 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[1] 1769073603.306247 write 0x8d909b thru 0x8e4d24 (0xbc8a bytes) 000005[1] 1769073603.358817 write 0x7ac11d thru 0x7b4da9 (0x8c8d bytes) 000006[0] 1769073603.465951 mapread 0x01e1cd thru 0x02ca0f (0xe843 bytes) 000008[1] 1769073603.633224 write 0x17243b thru 0x177917 (0x54dd bytes) truncating to largest ever: 0x27e355 000009[0] 1769073603.696279 trunc from 0x8e4d25 to 0x27e354 (0x6669d0 bytes) 000010[0] 1769073603.806414 read 0x134a5c thru 0x1364a1 (0x1a46 bytes) 000012[1] 1769073603.861042 mapwrite 0x90bf9c thru 0x9113e1 (0x5446 bytes) 000015[1] 1769073604.172235 mapwrite 0x7b6c6b thru 0x7c345f (0xc7f5 bytes) 000018[0] 1769073604.417960 mapread 0x877dc4 thru 0x87eecb (0x7108 bytes) 000020[0] 1769073604.595064 mapread 0x235457 thru 0x242424 (0xcfce bytes) 000021[0] 1769073604.666243 write 0x5a6579 thru 0x5b546e (0xeef6 bytes) 000023[0] 1769073604.741229 mapread 0x826f32 thru 0x8366ce (0xf79d bytes) 000024[1] 1769073604.775600 read 0x3efead thru 0x3fd57a (0xd6ce bytes) 000025[1] 1769073604.804203 write 0x0ec750 thru 0x0f7000 (0xa8b1 bytes) 000026[0] 1769073604.861473 read 0x41e4bb thru 0x41f397 (0xedd bytes) 000030[0] 1769073604.939847 mapread 0x2f7c90 thru 0x305feb (0xe35c bytes) 000031[1] 1769073605.028272 mapread 0x2f5607 thru 0x2f8975 (0x336f bytes) 000032[1] 1769073605.150890 mapwrite 0x97e58d thru 0x98cc6c (0xe6e0 bytes) 000033[1] 1769073605.459447 write 0x61c32c thru 0x61cf1d (0xbf2 bytes) 000034[1] 1769073605.469788 read 0x094e3c thru 0x09c9d3 (0x7b98 bytes) 000035[1] 1769073605.556042 mapread 0x1c0780 thru 0x1c2705 (0x1f86 bytes) 000037[1] 1769073605.664612 mapwrite 0x39ca09 thru 0x3a1218 (0x4810 bytes) 000038[1] 1769073605.880418 read 0x4d22bd thru 0x4d50f5 (0x2e39 bytes) 000043[1] 1769073606.272070 mapread 0x3f5e05 thru 0x4035bd (0xd7b9 bytes) truncating to largest ever: 0x415c98 000048[0] 1769073606.468843 trunc from 0x98cc6d to 0x415c97 (0x576fd5 bytes) 000051[1] 1769073606.688269 read 0x3ba73e thru 0x3c8ed6 (0xe799 bytes) truncating to largest ever: 0x4a9497 000052[0] 1769073606.821777 trunc from 0x415c98 to 0x4a9496 (0x937ff bytes) 000054[0] 1769073606.953520 read 0x22a37a thru 0x23427c (0x9f03 bytes) 000057[0] 1769073606.976646 read 0x2a0cdb thru 0x2a6a1b (0x5d41 bytes) 000058[1] 1769073606.997299 mapread 0x191ce6 thru 0x1994f7 (0x7812 bytes) 000059[1] 1769073607.106944 mapwrite 0x24cc35 thru 0x2521e6 (0x55b2 bytes) 000060[0] 1769073607.382925 write 0x5d7de4 thru 0x5e6984 (0xeba1 bytes) truncating to largest ever: 0x5e0c96 000062[0] 1769073607.453492 trunc from 0x5e6985 to 0x5e0c95 (0x5cef bytes) 000063[0] 1769073607.514173 read 0x4d4a57 thru 0x4e47c9 (0xfd73 bytes) 000066[0] 1769073607.593241 read 0x02f67c thru 0x03decc (0xe851 bytes) 000067[0] 1769073607.606749 write 0x8a3207 thru 0x8a52e3 (0x20dd bytes) 000070[0] 1769073607.665040 write 0x47803a thru 0x481c32 (0x9bf9 bytes) 000071[1] 1769073607.720949 write 0x29fe80 thru 0x2a10ef (0x1270 bytes) 000072[0] 1769073607.813110 read 0x2eb631 thru 0x2ee94e (0x331e bytes) 000077[1] 1769073607.931004 mapread 0x003ce0 thru 0x00c044 (0x8365 bytes) 000078[0] 1769073608.026802 mapread 0x83680d thru 0x84140f (0xac03 bytes) 000080[1] 1769073608.072356 write 0x8fa5cd thru 0x905a9b (0xb4cf bytes) 000082[0] 1769073608.179687 mapread 0x088aa6 thru 0x08bca5 (0x3200 bytes) 000085[0] 1769073608.388918 trunc from 0x905a9c to 0x50ca5f (0x3f903c bytes) 000086[1] 1769073608.454528 write 0x3aee85 thru 0x3b5fce (0x714a bytes) 000090[1] 1769073608.724791 mapwrite 0x5b27cd thru 0x5b7925 (0x5159 bytes) 000092[0] 1769073609.104498 mapwrite 0x0ff249 thru 0x10677d (0x7535 bytes) 000093[1] 1769073609.397143 mapread 0x3bc748 thru 0x3cb8f5 (0xf1ae bytes) 000096[1] 1769073609.665903 trunc from 0x5b7926 to 0x055c2d (0x561cf8 bytes) 000097[0] 1769073609.714400 mapwrite 0x69eeeb thru 0x6a480e (0x5924 bytes) 000099[0] 1769073609.878298 mapread 0x4bbc33 thru 0x4c3f76 (0x8344 bytes) 000100[0] 1769073609.939228 mapread 0x14e2f7 thru 0x15a755 (0xc45f bytes) 000101[1] 1769073609.997253 mapread 0x0393d7 thru 0x03dc24 (0x484e bytes) 000104[0] 1769073610.217748 write 0x21a9ca thru 0x22a8fb (0xff32 bytes) 000106[1] 1769073610.427653 mapread 0x18bde4 thru 0x194f68 (0x9185 bytes) 000107[1] 1769073610.621651 write 0x76538e thru 0x769a1f (0x4692 bytes) 000111[1] 1769073610.749736 mapwrite 0x8f5ffa thru 0x9017e9 (0xb7f0 bytes) 000112[1] 1769073611.166498 write 0x66e4eb thru 0x66f5f4 (0x110a bytes) truncating to largest ever: 0x69e6d4 000115[1] 1769073611.301976 trunc from 0x9017ea to 0x69e6d3 (0x263116 bytes) 000116[1] 1769073611.384901 read 0x51b278 thru 0x522ea5 (0x7c2e bytes) 000117[1] 1769073611.514823 mapwrite 0x031985 thru 0x033d52 (0x23ce bytes) 000119[0] 1769073611.842069 mapwrite 0x7fefa8 thru 0x80a6b4 (0xb70d bytes) 000120[1] 1769073612.056378 mapwrite 0x1ef418 thru 0x1f9a10 (0xa5f9 bytes) 000126[1] 1769073612.395139 mapread 0x7aa1aa thru 0x7aacc8 (0xb1f bytes) 000129[1] 1769073612.436143 write 0x5b1ce2 thru 0x5befc6 (0xd2e5 bytes) 000130[1] 1769073612.496865 mapread 0x3788f6 thru 0x37fe56 (0x7561 bytes) 000136[0] 1769073612.572050 mapread 0x7b133a thru 0x7b66dc (0x53a3 bytes) 000137[1] 1769073612.659106 mapread 0x10a8bb thru 0x1139d6 (0x911c bytes) 000138[1] 1769073612.693607 trunc from 0x80a6b5 to 0x189b4e (0x680b66 bytes) 000143[1] 1769073612.770124 read 0x17aefc thru 0x180389 (0x548e bytes) 000144[1] 1769073612.829293 read 0x03af4a thru 0x03bac7 (0xb7e bytes) 000145[1] 1769073612.848916 write 0x881774 thru 0x887425 (0x5cb2 bytes) 000146[0] 1769073612.878160 mapwrite 0x73d50e thru 0x73dcf8 (0x7eb bytes) 000147[0] 1769073612.973418 read 0x77ecb3 thru 0x780176 (0x14c4 bytes) 000149[0] 1769073613.035506 write 0x636e66 thru 0x63e24a (0x73e5 bytes) 000150[1] 1769073613.060167 mapread 0x0314f6 thru 0x03c6cf (0xb1da bytes) 000152[1] 1769073613.098222 mapwrite 0x21cf2e thru 0x221483 (0x4556 bytes) 000154[1] 1769073613.231138 trunc from 0x887426 to 0x17dde9 (0x70963c bytes) 000158[1] 1769073613.333823 trunc from 0x17ddea to 0x66ee96 (0x4f10ad bytes) 000161[0] 1769073613.397896 mapread 0x1ecf7f thru 0x1f669a (0x971c bytes) 000162[1] 1769073613.448232 read 0x11e6de thru 0x1238e7 (0x520a bytes) 000163[1] 1769073613.455583 trunc from 0x66ee97 to 0x54709a (0x127dfc bytes) 000166[0] 1769073613.561450 mapwrite 0x19d06e thru 0x19f66f (0x2602 bytes) 000167[1] 1769073613.671307 mapwrite 0x9320d8 thru 0x938deb (0x6d14 bytes) 000169[1] 1769073613.785316 mapread 0x4819d2 thru 0x490677 (0xeca6 bytes) 000170[0] 1769073613.836021 mapwrite 0x5aa42f thru 0x5b1c50 (0x7822 bytes) 000172[0] 1769073613.968649 mapread 0x90a24b thru 0x910692 (0x6448 bytes) 000173[1] 1769073614.007114 mapread 0x069eff thru 0x06ab80 (0xc82 bytes) 000175[0] 1769073614.030778 mapwrite 0x12df2a thru 0x1386ce (0xa7a5 bytes) 000178[0] 1769073614.181865 trunc from 0x938dec to 0x0165e3 (0x922808 bytes) 000179[1] 1769073614.226415 mapwrite 0x471ae5 thru 0x47445a (0x2976 bytes) 000181[1] 1769073614.334189 mapread 0x387903 thru 0x3883eb (0xae9 bytes) 000182[1] 1769073614.374061 mapread 0x18c8a3 thru 0x190618 (0x3d76 bytes) 000183[0] 1769073614.415046 write 0x7b6ddb thru 0x7c4074 (0xd29a bytes) 000186[0] 1769073614.474029 mapwrite 0x016465 thru 0x019a8a (0x3626 bytes) 000189[0] 1769073614.760859 write 0x40c2c2 thru 0x41957a (0xd2b9 bytes) 000191[1] 1769073614.877169 mapread 0x4fc328 thru 0x502e70 (0x6b49 bytes) 000192[1] 1769073614.981969 write 0x2f4241 thru 0x2f535b (0x111b bytes) 000193[1] 1769073615.017999 trunc from 0x7c4075 to 0x3f6449 (0x3cdc2b bytes) 000194[1] 1769073615.089402 mapread 0x236ca6 thru 0x2377ca (0xb25 bytes) 000196[1] 1769073615.111252 read 0x26a8a7 thru 0x26eed4 (0x462e bytes) 000197[0] 1769073615.273717 write 0x4b1915 thru 0x4c0d8f (0xf47b bytes) 000199[0] 1769073615.335926 mapwrite 0x5207d3 thru 0x52d9ea (0xd218 bytes) 000202[0] 1769073615.644451 mapwrite 0x642b81 thru 0x645cb9 (0x3139 bytes) 000203[1] 1769073615.906073 mapread 0x01132f thru 0x01a18d (0x8e5f bytes) truncating to largest ever: 0x85f95c 000207[1] 1769073616.167282 trunc from 0x645cba to 0x85f95b (0x219ca2 bytes) 000208[1] 1769073616.266610 mapwrite 0x8d8f47 thru 0x8e5a77 (0xcb31 bytes) 000211[1] 1769073616.665331 mapwrite 0x43df93 thru 0x449b9c (0xbc0a bytes) 000218[1] 1769073616.883226 mapread 0x8b1632 thru 0x8b2161 (0xb30 bytes) 000219[1] 1769073616.929996 mapread 0x487cc2 thru 0x489e80 (0x21bf bytes) 000220[1] 1769073616.933159 trunc from 0x8e5a78 to 0x24dd28 (0x697d4f bytes) 000221[0] 1769073617.003305 mapread 0x13a8fd thru 0x14a608 (0xfd0c bytes) 000225[0] 1769073617.159391 mapread 0x1da4fe thru 0x1da7ad (0x2b0 bytes) 000227[0] 1769073617.181796 mapwrite 0x949dab thru 0x957e90 (0xe0e6 bytes) 000228[0] 1769073617.339147 read 0x5fba5e thru 0x5fec31 (0x31d4 bytes) 000230[1] 1769073617.377675 write 0x11ccef thru 0x1212ad (0x45bf bytes) 000233[0] 1769073617.450362 mapread 0x312eb1 thru 0x31f4b8 (0xc608 bytes) 000236[0] 1769073617.537143 read 0x118225 thru 0x120338 (0x8114 bytes) 000238[0] 1769073617.547185 write 0x448f53 thru 0x451a2c (0x8ada bytes) 000239[0] 1769073617.568848 read 0x37312d thru 0x37c775 (0x9649 bytes) 000241[0] 1769073617.597118 write 0x40931c thru 0x40d7fd (0x44e2 bytes) 000244[0] 1769073617.620842 mapwrite 0x2e2f1c thru 0x2ecc88 (0x9d6d bytes) 000245[0] 1769073617.764783 write 0x7d2f0a thru 0x7e204a (0xf141 bytes) truncating to largest ever: 0x948178 000247[1] 1769073617.804005 trunc from 0x957e91 to 0x948177 (0xfd19 bytes) 000249[0] 1769073617.941014 trunc from 0x948178 to 0x116a39 (0x83173e bytes) 000250[1] 1769073618.004113 mapread 0x01305c thru 0x01a4fc (0x74a1 bytes) 000253[0] 1769073618.114867 mapread 0x0fe0ae thru 0x105037 (0x6f8a bytes) 000254[0] 1769073618.135792 trunc from 0x116a3a to 0x45bfd3 (0x34559a bytes) 000256[1] 1769073618.204232 read 0x2deff8 thru 0x2eb215 (0xc21e bytes) 000258[0] 1769073618.258659 trunc from 0x45bfd4 to 0x5db818 (0x17f845 bytes) 000260[1] 1769073618.330355 read 0x4efa71 thru 0x4f8c9d (0x922d bytes) 000263[0] 1769073618.393845 read 0x44df43 thru 0x45dbd6 (0xfc94 bytes) 000266[0] 1769073618.428131 read 0x1350ab thru 0x138f38 (0x3e8e bytes) 000269[0] 1769073618.450796 write 0x70a800 thru 0x7195b6 (0xedb7 bytes) 000270[1] 1769073618.474126 mapread 0x02a8a0 thru 0x02cf6f (0x26d0 bytes) 000272[1] 1769073618.567475 trunc from 0x7195b7 to 0x126030 (0x5f3586 bytes) 000273[1] 1769073618.667240 read 0x10e068 thru 0x11cd56 (0xecef bytes) 000274[0] 1769073618.807631 write 0x626350 thru 0x62ccad (0x695e bytes) 000277[1] 1769073618.895794 trunc from 0x62ccae to 0x2aeda5 (0x37df08 bytes) 000278[1] 1769073618.938309 mapread 0x1fa5af thru 0x2061bc (0xbc0e bytes) 000279[0] 1769073618.988407 read 0x1adba0 thru 0x1bd060 (0xf4c1 bytes) 000280[0] 1769073619.040272 mapwrite 0x8c490d thru 0x8c5399 (0xa8d bytes) 000281[1] 1769073619.193901 mapwrite 0x8b0220 thru 0x8b2abe (0x289f bytes) 000286[1] 1769073619.437323 mapread 0x4907c3 thru 0x499aa4 (0x92e2 bytes) 000288[1] 1769073619.609437 write 0x176a0f thru 0x180589 (0x9b7b bytes) 000291[1] 1769073619.639215 read 0x519bff thru 0x527235 (0xd637 bytes) 000294[0] 1769073619.664256 mapwrite 0x1b9054 thru 0x1c8f3d (0xfeea bytes) 000297[1] 1769073619.818341 mapread 0x2799f5 thru 0x2800aa (0x66b6 bytes) 000300[1] 1769073619.901290 read 0x5b53af thru 0x5c3169 (0xddbb bytes) 000302[0] 1769073619.902350 read 0x2604c8 thru 0x26a02e (0x9b67 bytes) 000303[0] 1769073619.910225 mapwrite 0x93ef53 thru 0x93f1ed (0x29b bytes) 000304[0] 1769073620.028944 mapwrite 0x56a773 thru 0x57873d (0xdfcb bytes) 000306[0] 1769073620.108833 write 0x929a7a thru 0x932f32 (0x94b9 bytes) 000307[0] 1769073620.145900 mapwrite 0x0f8ea9 thru 0x0fecc4 (0x5e1c bytes) 000309[0] 1769073620.269469 mapread 0x116ab7 thru 0x11d4b0 (0x69fa bytes) 000315[0] 1769073620.378241 mapwrite 0x6f018c thru 0x6ffc06 (0xfa7b bytes) 000317[1] 1769073620.455000 trunc from 0x93f1ee to 0x123bc2 (0x81b62b bytes) 000318[0] 1769073620.504276 mapwrite 0x394456 thru 0x397dfc (0x39a7 bytes) 000320[0] 1769073620.684848 read 0x153003 thru 0x160fce (0xdfcc bytes) 000321[0] 1769073620.878583 trunc from 0x397dfd to 0x80fc5b (0x477e5f bytes) 000322[1] 1769073620.972215 mapread 0x50207b thru 0x503201 (0x1187 bytes) 000323[1] 1769073621.070864 trunc from 0x80fc5c to 0x0e2033 (0x72dc28 bytes) 000324[1] 1769073621.198760 trunc from 0x0e2034 to 0x53bf9c (0x459f69 bytes) 000325[0] 1769073621.332280 mapread 0x167b0a thru 0x16bf69 (0x4460 bytes) 000326[1] 1769073621.463079 write 0x392639 thru 0x3993e2 (0x6daa bytes) 000329[1] 1769073621.520137 read 0x42dec5 thru 0x43aaeb (0xcc27 bytes) 000332[1] 1769073621.548627 trunc from 0x53bf9d to 0x24f00d (0x2ecf8f bytes) 000333[0] 1769073621.570140 trunc from 0x24f00e to 0x53976b (0x2ea75e bytes) 000336[1] 1769073621.625312 mapwrite 0x66191e thru 0x66f659 (0xdd3c bytes) 000338[0] 1769073621.903829 mapwrite 0x869dd8 thru 0x86f7b6 (0x59df bytes) 000339[1] 1769073622.239227 trunc from 0x86f7b7 to 0x8163f9 (0x593bd bytes) 000340[1] 1769073622.374926 write 0x2a585f thru 0x2aac45 (0x53e7 bytes) 000343[1] 1769073622.472789 trunc from 0x8163fa to 0x0d4ca6 (0x741753 bytes) 000346[1] 1769073622.629838 mapwrite 0x84876d thru 0x84c1e8 (0x3a7c bytes) 000347[1] 1769073622.849014 read 0x65a933 thru 0x669000 (0xe6ce bytes) 000348[1] 1769073622.884644 mapwrite 0x3ab8b3 thru 0x3b018f (0x48dd bytes) 000351[1] 1769073622.989166 read 0x2f97fa thru 0x30625d (0xca64 bytes) 000352[1] 1769073623.003854 trunc from 0x84c1e9 to 0x58a51c (0x2c1ccc bytes) 000355[0] 1769073623.080241 mapwrite 0x99c540 thru 0x9abfd9 (0xfa9a bytes) 000357[0] 1769073623.267375 mapread 0x790540 thru 0x79e215 (0xdcd6 bytes) 000359[0] 1769073623.541204 mapread 0x730400 thru 0x73a704 (0xa305 bytes) 000361[1] 1769073623.681936 write 0x0f82c3 thru 0x100258 (0x7f96 bytes) 000363[0] 1769073623.777880 write 0x705827 thru 0x7074cf (0x1ca9 bytes) 000365[1] 1769073623.816954 read 0x898f42 thru 0x8a1507 (0x85c6 bytes) 000367[0] 1769073623.841915 trunc from 0x9abfda to 0x49d01c (0x50efbd bytes) 000368[0] 1769073623.884794 mapwrite 0x2c1818 thru 0x2c66da (0x4ec3 bytes) 000370[0] 1769073623.951277 mapread 0x000435 thru 0x00eb4a (0xe716 bytes) 000372[1] 1769073624.018293 mapread 0x394a2f thru 0x3a4385 (0xf957 bytes) 000374[1] 1769073624.081428 trunc from 0x49d01d to 0x269356 (0x233cc6 bytes) 000378[1] 1769073624.145055 write 0x830001 thru 0x83e685 (0xe685 bytes) 000379[0] 1769073624.159303 write 0x98971a thru 0x991ffc (0x88e3 bytes) 000383[1] 1769073624.293492 write 0x42f783 thru 0x43a56e (0xadec bytes) 000384[0] 1769073624.308038 read 0x1316d7 thru 0x13923f (0x7b69 bytes) 000385[0] 1769073624.321527 trunc from 0x991ffd to 0x4d21ed (0x4bfe0f bytes) 000386[1] 1769073624.355690 mapread 0x13ea44 thru 0x14cb75 (0xe132 bytes) 000391[0] 1769073624.448011 trunc from 0x4d21ee to 0x0ab9bb (0x426832 bytes) 000392[0] 1769073624.508343 trunc from 0x0ab9bc to 0x3e25bb (0x336c00 bytes) 000396[1] 1769073624.700076 read 0x1ad724 thru 0x1b3c98 (0x6575 bytes) 000397[1] 1769073624.812726 trunc from 0x3e25bc to 0x70d1cf (0x32ac14 bytes) 000398[1] 1769073624.867229 trunc from 0x70d1d0 to 0x709dac (0x3423 bytes) 000400[1] 1769073624.903284 read 0x69319c thru 0x694079 (0xede bytes) 000403[0] 1769073624.912782 mapread 0x432103 thru 0x43be2f (0x9d2d bytes) 000407[0] 1769073625.001256 write 0x101acd thru 0x10b305 (0x9839 bytes) 000408[0] 1769073625.029068 mapread 0x243794 thru 0x244115 (0x982 bytes) 000409[1] 1769073625.039080 trunc from 0x709dad to 0x5de270 (0x12bb3c bytes) 000410[1] 1769073625.130944 trunc from 0x5de271 to 0x918512 (0x33a2a2 bytes) 000417[0] 1769073625.317018 mapwrite 0x60c593 thru 0x60f9ff (0x346d bytes) 000418[0] 1769073625.495086 mapwrite 0x16b0a2 thru 0x179be1 (0xeb40 bytes) 000419[0] 1769073625.779031 mapwrite 0x0f3b2e thru 0x0f6744 (0x2c17 bytes) 000421[0] 1769073625.909517 mapread 0x7b9dff thru 0x7bdfa8 (0x41aa bytes) 000422[1] 1769073625.983231 mapread 0x661e40 thru 0x6643ca (0x258b bytes) 000423[0] 1769073626.064062 mapwrite 0x948665 thru 0x94e554 (0x5ef0 bytes) 000424[1] 1769073626.362878 read 0x1af994 thru 0x1b4455 (0x4ac2 bytes) 000426[1] 1769073626.512035 trunc from 0x94e555 to 0x338a8a (0x615aca bytes) 000427[0] 1769073626.671162 mapread 0x0c3a62 thru 0x0c5b97 (0x2136 bytes) 000429[1] 1769073626.835719 mapwrite 0x2ccd9e thru 0x2d3b55 (0x6db8 bytes) 000431[1] 1769073627.096166 read 0x0b9912 thru 0x0c755b (0xdc4a bytes) 000436[0] 1769073627.172057 trunc from 0x338a8b to 0x8e99e0 (0x5b0f56 bytes) 000437[1] 1769073627.237662 mapread 0x609746 thru 0x6110a4 (0x795f bytes) 000438[0] 1769073627.314062 mapread 0x6437c4 thru 0x64fb3d (0xc37a bytes) 000439[1] 1769073627.340978 read 0x397b25 thru 0x3a4741 (0xcc1d bytes) 000441[1] 1769073627.374308 mapread 0x79acb2 thru 0x7aa307 (0xf656 bytes) 000443[0] 1769073627.411995 mapwrite 0x5ad0c4 thru 0x5bbea2 (0xeddf bytes) 000444[0] 1769073627.530129 read 0x27dd6f thru 0x2844c2 (0x6754 bytes) 000445[1] 1769073627.570012 read 0x0f5fa9 thru 0x103a27 (0xda7f bytes) 000447[1] 1769073627.607905 mapread 0x298d49 thru 0x29acf9 (0x1fb1 bytes) 000448[1] 1769073627.657457 mapwrite 0x41f227 thru 0x427a9c (0x8876 bytes) 000449[1] 1769073627.918683 mapread 0x065667 thru 0x071132 (0xbacc bytes) 000450[1] 1769073628.053905 trunc from 0x8e99e1 to 0x2c6893 (0x62314d bytes) 000454[0] 1769073628.175816 trunc from 0x2c6894 to 0x6c7e50 (0x4015bd bytes) 000458[1] 1769073628.327523 trunc from 0x6c7e51 to 0x84a990 (0x182b40 bytes) 000459[0] 1769073628.414072 read 0x1af95a thru 0x1b183f (0x1ee6 bytes) 000460[1] 1769073628.443512 write 0x131538 thru 0x13710e (0x5bd7 bytes) 000461[1] 1769073628.498750 mapwrite 0x923d7e thru 0x9253ce (0x1651 bytes) 000462[0] 1769073628.693760 mapwrite 0x3014bb thru 0x307b6a (0x66b0 bytes) 000464[1] 1769073628.921670 mapread 0x27d3be thru 0x28bb18 (0xe75b bytes) 000465[1] 1769073629.039800 write 0x848e60 thru 0x85436d (0xb50e bytes) 000467[0] 1769073629.085346 mapwrite 0x9b618a thru 0x9bcd13 (0x6b8a bytes) 000470[1] 1769073629.421198 write 0x94bb88 thru 0x94fd34 (0x41ad bytes) 000473[1] 1769073629.542215 write 0x1d80ed thru 0x1e4ea2 (0xcdb6 bytes) 000474[1] 1769073629.575334 write 0x24799d thru 0x25257d (0xabe1 bytes) 000475[0] 1769073629.648756 read 0x44109e thru 0x44630e (0x5271 bytes) 000476[1] 1769073629.728431 mapread 0x71b77c thru 0x7246ff (0x8f84 bytes) 000478[1] 1769073629.853356 write 0x30552f thru 0x30e54b (0x901d bytes) 000480[1] 1769073629.873492 read 0x7a64f4 thru 0x7b4b64 (0xe671 bytes) 000481[0] 1769073629.890919 write 0x7d6cdc thru 0x7da638 (0x395d bytes) 000487[1] 1769073630.191908 read 0x6492a3 thru 0x6572be (0xe01c bytes) 000488[0] 1769073630.249990 write 0x65ad30 thru 0x65f59a (0x486b bytes) 000490[0] 1769073630.346519 write 0x8f58f8 thru 0x8fe7c8 (0x8ed1 bytes) 000495[0] 1769073630.536674 write 0x68e864 thru 0x695b05 (0x72a2 bytes) 000496[1] 1769073630.551144 mapwrite 0x9867ab thru 0x98f5dc (0x8e32 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3630 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1769073631.618783 write 0x82a000 thru 0x835037 (0xb038 bytes) 000002[1] 1769073631.640195 write 0x3b1000 thru 0x3bf40a (0xe40b bytes) 000003[1] 1769073631.726984 read 0x623000 thru 0x626aa9 (0x3aaa bytes) truncating to largest ever: 0x542dd6 000004[0] 1769073631.738561 trunc from 0x835038 to 0x542dd5 (0x2f2262 bytes) 000006[0] 1769073631.792122 write 0x7ff000 thru 0x803990 (0x4991 bytes) 000008[1] 1769073631.831291 write 0x644000 thru 0x649c66 (0x5c67 bytes) 000009[0] 1769073631.865814 write 0x5d1000 thru 0x5e0de3 (0xfde4 bytes) 000011[1] 1769073631.942084 read 0x0d6000 thru 0x0e32ba (0xd2bb bytes) 000012[0] 1769073631.968339 write 0x6a0000 thru 0x6a8a37 (0x8a38 bytes) 000015[0] 1769073631.993176 mapread 0x7e7000 thru 0x7ed97c (0x697d bytes) 000016[1] 1769073632.027831 write 0x5d5000 thru 0x5d8fa4 (0x3fa5 bytes) truncating to largest ever: 0x679b12 000020[1] 1769073632.090375 trunc from 0x803991 to 0x679b11 (0x189e7f bytes) 000023[0] 1769073632.125971 mapwrite 0x6c1000 thru 0x6cef2a (0xdf2b bytes) 000024[1] 1769073632.257132 mapread 0x199000 thru 0x19c314 (0x3315 bytes) 000025[1] 1769073632.288658 mapread 0x00b000 thru 0x013545 (0x8546 bytes) truncating to largest ever: 0x88d384 000031[1] 1769073632.312741 trunc from 0x6cef2b to 0x88d383 (0x1be459 bytes) 000032[0] 1769073632.336894 mapwrite 0x6e2000 thru 0x6eeb96 (0xcb97 bytes) 000033[1] 1769073632.460110 write 0x6a6000 thru 0x6afcc3 (0x9cc4 bytes) 000036[1] 1769073632.519275 read 0x7f2000 thru 0x7f21ed (0x1ee bytes) 000037[0] 1769073632.533940 mapread 0x415000 thru 0x417612 (0x2613 bytes) 000039[1] 1769073632.575898 mapread 0x330000 thru 0x333328 (0x3329 bytes) 000040[0] 1769073632.588389 read 0x200000 thru 0x20ad46 (0xad47 bytes) 000052[1] 1769073632.597946 trunc from 0x88d384 to 0x7fc525 (0x90e5e bytes) 000053[0] 1769073632.625658 trunc from 0x7fc526 to 0x299aef (0x562a36 bytes) 000058[0] 1769073632.670306 mapwrite 0x289000 thru 0x28ba7f (0x2a80 bytes) 000065[0] 1769073632.767926 mapwrite 0x69c000 thru 0x6a59ba (0x99bb bytes) 000068[0] 1769073632.879410 write 0x380000 thru 0x383cf0 (0x3cf1 bytes) 000069[0] 1769073632.885323 mapwrite 0x025000 thru 0x0306e5 (0xb6e6 bytes) 000072[0] 1769073633.059251 read 0x34b000 thru 0x358a87 (0xda88 bytes) 000074[0] 1769073633.070936 trunc from 0x6a59bb to 0x7527e2 (0xace28 bytes) 000075[0] 1769073633.103387 mapwrite 0x22e000 thru 0x232c81 (0x4c82 bytes) 000077[0] 1769073633.192006 read 0x092000 thru 0x09db1b (0xbb1c bytes) 000079[0] 1769073633.227204 mapwrite 0x194000 thru 0x1a2044 (0xe045 bytes) 000082[1] 1769073633.353469 write 0x354000 thru 0x3572af (0x32b0 bytes) 000084[0] 1769073633.376856 mapread 0x3f5000 thru 0x3ffff6 (0xaff7 bytes) 000087[0] 1769073633.426287 mapwrite 0x67a000 thru 0x683873 (0x9874 bytes) 000088[0] 1769073633.501356 write 0x513000 thru 0x5224d4 (0xf4d5 bytes) 000089[1] 1769073633.549960 write 0x1f8000 thru 0x1fb125 (0x3126 bytes) 000090[0] 1769073633.629202 trunc from 0x7527e3 to 0x299e05 (0x4b89dd bytes) 000093[0] 1769073633.736835 read 0x23e000 thru 0x2469b6 (0x89b7 bytes) 000095[0] 1769073633.917407 mapwrite 0x2fe000 thru 0x30492f (0x6930 bytes) 000098[1] 1769073634.238404 mapread 0x262000 thru 0x270ea8 (0xeea9 bytes) 000101[1] 1769073634.291309 read 0x1db000 thru 0x1e6bd9 (0xbbda bytes) 000108[1] 1769073634.394144 mapread 0x104000 thru 0x11024c (0xc24d bytes) 000111[0] 1769073634.396822 mapwrite 0x82a000 thru 0x8366e3 (0xc6e4 bytes) 000112[0] 1769073634.683018 mapwrite 0x513000 thru 0x51e257 (0xb258 bytes) 000116[0] 1769073634.813748 write 0x6f3000 thru 0x6fab51 (0x7b52 bytes) 000117[1] 1769073634.850910 mapwrite 0x847000 thru 0x853244 (0xc245 bytes) 000118[1] 1769073635.276502 read 0x096000 thru 0x09f785 (0x9786 bytes) 000119[1] 1769073635.288105 read 0x5c5000 thru 0x5c8889 (0x388a bytes) 000122[1] 1769073635.338904 read 0x48a000 thru 0x499614 (0xf615 bytes) 000123[0] 1769073635.367477 trunc from 0x853245 to 0x25966e (0x5f9bd6 bytes) 000124[0] 1769073635.436779 mapread 0x05f000 thru 0x062efd (0x3efe bytes) 000129[1] 1769073635.595921 write 0x71b000 thru 0x7271b9 (0xc1ba bytes) 000131[1] 1769073635.634034 mapwrite 0x348000 thru 0x34b825 (0x3826 bytes) 000132[1] 1769073635.712508 mapread 0x613000 thru 0x6176b6 (0x46b7 bytes) 000134[1] 1769073635.737780 mapread 0x08d000 thru 0x08d5c7 (0x5c8 bytes) 000135[0] 1769073635.745512 mapwrite 0x2a8000 thru 0x2b4148 (0xc149 bytes) 000136[1] 1769073635.905821 mapread 0x1dd000 thru 0x1de4f9 (0x14fa bytes) 000137[0] 1769073635.946295 write 0x852000 thru 0x8532bc (0x12bd bytes) 000148[1] 1769073636.056099 mapwrite 0x281000 thru 0x28b6dc (0xa6dd bytes) 000149[1] 1769073636.238132 write 0x5b3000 thru 0x5bedee (0xbdef bytes) 000150[0] 1769073636.279200 read 0x56d000 thru 0x571c55 (0x4c56 bytes) 000154[0] 1769073636.328261 write 0x43a000 thru 0x447f35 (0xdf36 bytes) 000155[1] 1769073636.375779 trunc from 0x8532bd to 0x48effc (0x3c42c0 bytes) 000159[1] 1769073636.432867 mapwrite 0x113000 thru 0x11a91f (0x7920 bytes) 000161[1] 1769073636.472448 mapwrite 0x46e000 thru 0x471038 (0x3039 bytes) 000162[0] 1769073636.512088 mapread 0x359000 thru 0x35ae5d (0x1e5e bytes) 000163[1] 1769073636.533056 trunc from 0x48effd to 0x0d6c70 (0x3b838c bytes) 000164[0] 1769073636.552512 write 0x41a000 thru 0x42952b (0xf52c bytes) 000165[0] 1769073636.585093 write 0x3d8000 thru 0x3e28f9 (0xa8fa bytes) 000167[0] 1769073636.612463 read 0x35e000 thru 0x360399 (0x239a bytes) 000168[0] 1769073636.619696 read 0x38e000 thru 0x394a78 (0x6a79 bytes) 000170[0] 1769073636.633205 mapwrite 0x855000 thru 0x85ec43 (0x9c44 bytes) 000171[1] 1769073636.756227 read 0x016000 thru 0x01ebe2 (0x8be3 bytes) 000172[0] 1769073636.784124 read 0x052000 thru 0x053f71 (0x1f72 bytes) 000176[1] 1769073636.798707 mapread 0x5e8000 thru 0x5ed996 (0x5997 bytes) 000177[0] 1769073636.817827 write 0x1cb000 thru 0x1cec35 (0x3c36 bytes) truncating to largest ever: 0x9fe708 000179[1] 1769073636.853829 trunc from 0x85ec44 to 0x9fe707 (0x19fac4 bytes) 000180[1] 1769073636.911502 mapwrite 0x162000 thru 0x16c216 (0xa217 bytes) 000181[0] 1769073637.003645 trunc from 0x9fe708 to 0x1ef7aa (0x80ef5d bytes) 000182[1] 1769073637.031073 mapread 0x031000 thru 0x03d714 (0xc715 bytes) 000185[0] 1769073637.088055 mapread 0x119000 thru 0x120c54 (0x7c55 bytes) 000189[1] 1769073637.103709 read 0x172000 thru 0x174ff5 (0x2ff6 bytes) 000192[1] 1769073637.146018 mapread 0x09a000 thru 0x0a8960 (0xe961 bytes) 000193[1] 1769073637.146889 trunc from 0x1ef7ab to 0x9ca8e3 (0x7db139 bytes) 000196[1] 1769073637.216889 mapread 0x8a8000 thru 0x8abc8a (0x3c8b bytes) 000197[1] 1769073637.400575 read 0x5e5000 thru 0x5ea263 (0x5264 bytes) 000200[0] 1769073637.417211 mapwrite 0x5e4000 thru 0x5e7a3c (0x3a3d bytes) 000202[0] 1769073637.558153 mapwrite 0x0f6000 thru 0x1003c9 (0xa3ca bytes) 000208[1] 1769073637.700317 trunc from 0x9ca8e4 to 0x552927 (0x477fbc bytes) 000209[1] 1769073637.751125 read 0x524000 thru 0x53018d (0xc18e bytes) 000211[0] 1769073637.763071 mapwrite 0x0b0000 thru 0x0b5cea (0x5ceb bytes) 000212[0] 1769073637.875934 trunc from 0x552928 to 0x2caffd (0x28792a bytes) 000214[0] 1769073637.926888 read 0x121000 thru 0x125ffc (0x4ffd bytes) 000218[1] 1769073638.002430 mapwrite 0x647000 thru 0x650fb3 (0x9fb4 bytes) 000220[1] 1769073638.172931 mapread 0x59f000 thru 0x5a7723 (0x8724 bytes) 000222[0] 1769073638.231247 mapwrite 0x929000 thru 0x92b85c (0x285d bytes) 000223[0] 1769073638.566301 mapwrite 0x503000 thru 0x50c00b (0x900c bytes) 000225[1] 1769073638.740105 mapread 0x746000 thru 0x7543fe (0xe3ff bytes) 000226[1] 1769073638.875449 write 0x3e7000 thru 0x3f32f2 (0xc2f3 bytes) 000229[1] 1769073638.930684 write 0x16c000 thru 0x16f757 (0x3758 bytes) 000232[0] 1769073639.003365 write 0x6cf000 thru 0x6d5f25 (0x6f26 bytes) 000233[1] 1769073639.112760 trunc from 0x92b85d to 0x70b5e3 (0x220279 bytes) 000234[0] 1769073639.217627 write 0x8b4000 thru 0x8bc863 (0x8864 bytes) 000235[0] 1769073639.309347 read 0x838000 thru 0x8479b5 (0xf9b6 bytes) 000237[1] 1769073639.358728 trunc from 0x8bc864 to 0x00747f (0x8b53e4 bytes) 000238[0] 1769073639.438974 write 0x7a8000 thru 0x7b3544 (0xb545 bytes) 000239[0] 1769073639.498983 trunc from 0x7b3545 to 0x20880a (0x5aad3a bytes) 000240[0] 1769073639.534958 mapread 0x146000 thru 0x15370b (0xd70c bytes) 000244[1] 1769073639.568882 read_OD 0x195000 thru 0x19ffff (0xb000 bytes) 000252[0] 1769073639.618721 write 0x2ad000 thru 0x2b8cfa (0xbcfb bytes) 000254[0] 1769073639.663351 mapread 0x22a000 thru 0x234259 (0xa25a bytes) 000255[1] 1769073639.693341 mapread 0x10b000 thru 0x10ca13 (0x1a14 bytes) 000258[0] 1769073639.767944 trunc from 0x2b8cfb to 0x2c0ec6 (0x81cc bytes) 000259[1] 1769073639.855096 read_OD 0x06e000 thru 0x06ffff (0x2000 bytes) 000264[0] 1769073639.891293 write 0x103000 thru 0x111876 (0xe877 bytes) 000265[0] 1769073639.937264 mapwrite 0x10c000 thru 0x1143ef (0x83f0 bytes) 000266[1] 1769073640.063480 mapread 0x08b000 thru 0x0971b0 (0xc1b1 bytes) 000267[1] 1769073640.133146 trunc from 0x2c0ec7 to 0x8a47d8 (0x5e3912 bytes) 000270[1] 1769073640.315431 mapwrite 0x6c0000 thru 0x6c373b (0x373c bytes) 000273[0] 1769073640.732789 mapread 0x6c4000 thru 0x6d224a (0xe24b bytes) 000275[0] 1769073640.885390 mapread 0x756000 thru 0x75a698 (0x4699 bytes) 000276[1] 1769073640.915113 write_OD 0x7f6000 thru 0x802fff (0xd000 bytes) 000277[0] 1769073640.996945 trunc from 0x8a47d9 to 0x704e63 (0x19f975 bytes) 000278[0] 1769073641.053411 read 0x03a000 thru 0x03f623 (0x5624 bytes) 000280[1] 1769073641.124617 write_OD 0x5c0000 thru 0x5cafff (0xb000 bytes) 000284[0] 1769073641.211705 trunc from 0x704e64 to 0x82084e (0x11b9eb bytes) 000285[0] 1769073641.269711 read 0x3d6000 thru 0x3e22ae (0xc2af bytes) 000290[1] 1769073641.401021 mapwrite 0x90f000 thru 0x91012e (0x112f bytes) 000291[1] 1769073641.604894 read_OD 0x001000 thru 0x00afff (0xa000 bytes) 000296[1] 1769073641.664884 mapread 0x842000 thru 0x842697 (0x698 bytes) 000297[0] 1769073641.670946 trunc from 0x91012f to 0x08bf2b (0x884203 bytes) 000298[0] 1769073641.739968 mapread 0x051000 thru 0x0567d7 (0x57d8 bytes) 000302[1] 1769073641.822004 write_OD 0x79f000 thru 0x7a1fff (0x3000 bytes) 000305[1] 1769073641.907881 trunc from 0x7a2000 to 0x6924f3 (0x10fb0c bytes) 000306[0] 1769073641.963171 write 0x3af000 thru 0x3b8882 (0x9883 bytes) 000307[0] 1769073642.003420 mapwrite 0x5bb000 thru 0x5bd66f (0x2670 bytes) 000308[1] 1769073642.094016 trunc from 0x6924f4 to 0x899a79 (0x207586 bytes) 000309[1] 1769073642.141087 read_OD 0x851000 thru 0x855fff (0x5000 bytes) 000310[0] 1769073642.165913 trunc from 0x899a7a to 0x658a15 (0x241064 bytes) 000313[1] 1769073642.251167 write_OD 0x7c9000 thru 0x7c9fff (0x1000 bytes) 000314[0] 1769073642.320998 write 0x84b000 thru 0x84e772 (0x3773 bytes) 000319[0] 1769073642.417808 mapread 0x413000 thru 0x422237 (0xf238 bytes) 000320[0] 1769073642.469236 mapread 0x0de000 thru 0x0e9d9e (0xbd9f bytes) 000321[1] 1769073642.536056 trunc from 0x84e773 to 0x1f6a56 (0x657d1c bytes) 000324[1] 1769073642.642207 mapwrite 0x084000 thru 0x08bc6d (0x7c6e bytes) 000326[0] 1769073642.772164 trunc from 0x1f6a57 to 0x89629c (0x69f846 bytes) 000329[0] 1769073642.842025 trunc from 0x89629d to 0x8e710a (0x50e6e bytes) 000333[1] 1769073642.888066 trunc from 0x8e710b to 0x3acb53 (0x53a5b7 bytes) 000335[1] 1769073642.937799 write_OD 0x5a7000 thru 0x5a8fff (0x2000 bytes) 000340[1] 1769073643.077178 trunc from 0x5a9000 to 0x4cac85 (0xde37a bytes) 000341[0] 1769073643.162074 read 0x034000 thru 0x03b2ad (0x72ae bytes) 000343[0] 1769073643.228584 mapwrite 0x518000 thru 0x522c18 (0xac19 bytes) 000344[0] 1769073643.480450 mapread 0x504000 thru 0x50c579 (0x857a bytes) 000345[1] 1769073643.497894 mapread 0x1b8000 thru 0x1bd88f (0x5890 bytes) 000346[0] 1769073643.570392 trunc from 0x522c19 to 0x101707 (0x421511 bytes) 000347[1] 1769073643.640283 trunc from 0x101708 to 0x71c1aa (0x61aaa3 bytes) 000351[1] 1769073643.857845 mapwrite 0x0be000 thru 0x0c4b87 (0x6b88 bytes) 000353[1] 1769073644.018065 read_OD 0x045000 thru 0x048fff (0x4000 bytes) 000355[0] 1769073644.051228 read 0x22b000 thru 0x22e1a4 (0x31a5 bytes) 000356[0] 1769073644.061116 mapwrite 0x120000 thru 0x125300 (0x5301 bytes) 000359[0] 1769073644.211915 trunc from 0x71c1ab to 0x29c4e3 (0x47fcc7 bytes) 000361[0] 1769073644.278903 trunc from 0x29c4e4 to 0x1cc0bd (0xd0426 bytes) 000363[1] 1769073644.315242 trunc from 0x1cc0be to 0x35bf65 (0x18fea8 bytes) 000365[0] 1769073644.405361 write 0x62c000 thru 0x63a38f (0xe390 bytes) 000366[1] 1769073644.499694 write_OD 0x421000 thru 0x425fff (0x5000 bytes) 000367[0] 1769073644.706260 mapwrite 0x0de000 thru 0x0e84a1 (0xa4a2 bytes) 000368[0] 1769073644.808731 write 0x3df000 thru 0x3e647a (0x747b bytes) 000371[0] 1769073644.860930 mapwrite 0x426000 thru 0x42b180 (0x5181 bytes) 000372[1] 1769073644.949857 mapread 0x4d9000 thru 0x4dd60f (0x4610 bytes) 000373[0] 1769073645.111877 write 0x71c000 thru 0x724d09 (0x8d0a bytes) 000375[0] 1769073645.190523 mapread 0x1bc000 thru 0x1c4f51 (0x8f52 bytes) 000376[1] 1769073645.305877 trunc from 0x724d0a to 0x9785d4 (0x2538cb bytes) 000379[0] 1769073645.415109 trunc from 0x9785d5 to 0x573f17 (0x4046bd bytes) 000381[0] 1769073645.567007 mapwrite 0x6ed000 thru 0x6f7e4f (0xae50 bytes) 000383[1] 1769073645.769877 mapread 0x407000 thru 0x40814f (0x1150 bytes) 000385[0] 1769073645.870788 mapwrite 0x575000 thru 0x57df2e (0x8f2f bytes) 000387[1] 1769073646.063225 mapwrite 0x147000 thru 0x151e59 (0xae5a bytes) 000388[0] 1769073646.286612 write 0x906000 thru 0x90b644 (0x5645 bytes) 000389[0] 1769073646.320092 mapread 0x585000 thru 0x594c6a (0xfc6b bytes) 000390[0] 1769073646.356943 read 0x6ef000 thru 0x6fb5ed (0xc5ee bytes) 000392[0] 1769073646.432661 read 0x43f000 thru 0x44defb (0xeefc bytes) 000394[0] 1769073646.498938 mapwrite 0x7a7000 thru 0x7af245 (0x8246 bytes) 000397[1] 1769073646.780100 mapwrite 0x7f7000 thru 0x7fbcd3 (0x4cd4 bytes) 000399[1] 1769073647.160952 mapwrite 0x369000 thru 0x372c7b (0x9c7c bytes) 000400[0] 1769073647.317423 write 0x2fd000 thru 0x302c2f (0x5c30 bytes) 000403[0] 1769073647.375582 read 0x7d5000 thru 0x7d9bc5 (0x4bc6 bytes) 000405[0] 1769073647.425178 read 0x666000 thru 0x666a44 (0xa45 bytes) 000411[1] 1769073647.485689 mapread 0x8a6000 thru 0x8afa4a (0x9a4b bytes) 000412[1] 1769073647.521494 mapread 0x4c7000 thru 0x4c7a63 (0xa64 bytes) 000416[1] 1769073647.527047 mapwrite 0x122000 thru 0x12412d (0x212e bytes) 000418[1] 1769073647.614763 mapwrite 0x2c7000 thru 0x2d5aa6 (0xeaa7 bytes) 000419[0] 1769073647.716669 write 0x0b0000 thru 0x0b2296 (0x2297 bytes) 000420[0] 1769073647.752192 write 0x494000 thru 0x49a48d (0x648e bytes) 000421[1] 1769073647.800363 mapread 0x714000 thru 0x72191d (0xd91e bytes) 000423[0] 1769073647.894204 mapread 0x587000 thru 0x591f16 (0xaf17 bytes) 000424[0] 1769073647.938076 mapwrite 0x350000 thru 0x359f72 (0x9f73 bytes) 000426[0] 1769073648.076980 mapread 0x504000 thru 0x51391d (0xf91e bytes) 000428[1] 1769073648.141258 mapwrite 0x2a1000 thru 0x2aee86 (0xde87 bytes) 000430[1] 1769073648.274777 read_OD 0x5b3000 thru 0x5b6fff (0x4000 bytes) 000432[0] 1769073648.304758 mapwrite 0x85d000 thru 0x86afac (0xdfad bytes) 000435[1] 1769073648.470914 read_OD 0x8a3000 thru 0x8aafff (0x8000 bytes) 000436[0] 1769073648.497003 mapread 0x085000 thru 0x0883a0 (0x33a1 bytes) 000437[1] 1769073648.525241 write_OD 0x8d9000 thru 0x8e2fff (0xa000 bytes) 000439[1] 1769073648.602404 mapwrite 0x9f3000 thru 0x9f72af (0x42b0 bytes) 000440[1] 1769073648.751442 write_OD 0x849000 thru 0x853fff (0xb000 bytes) 000442[0] 1769073648.859570 write 0x8d9000 thru 0x8e6461 (0xd462 bytes) 000443[1] 1769073648.893394 read_OD 0x041000 thru 0x043fff (0x3000 bytes) 000444[0] 1769073648.925174 mapwrite 0x678000 thru 0x682ebc (0xaebd bytes) 000447[0] 1769073648.998104 trunc from 0x9f72b0 to 0x27bac8 (0x77b7e7 bytes) 000448[1] 1769073649.020864 trunc from 0x27bac9 to 0x914ec7 (0x6993ff bytes) 000449[1] 1769073649.078973 trunc from 0x914ec8 to 0x33164c (0x5e387b bytes) 000452[0] 1769073649.139962 write 0x2bb000 thru 0x2caaa8 (0xfaa9 bytes) 000453[0] 1769073649.184248 write 0x0e5000 thru 0x0e6170 (0x1171 bytes) 000461[1] 1769073649.269124 mapread 0x2d0000 thru 0x2d2b0b (0x2b0c bytes) 000465[1] 1769073649.316299 read_OD 0x2a0000 thru 0x2a3fff (0x4000 bytes) 000467[1] 1769073649.329243 mapread 0x07c000 thru 0x080c5a (0x4c5b bytes) 000471[1] 1769073649.401626 mapread 0x07c000 thru 0x084844 (0x8845 bytes) 000472[0] 1769073649.402776 trunc from 0x33164d to 0x6bd083 (0x38ba37 bytes) 000473[0] 1769073649.447195 trunc from 0x6bd084 to 0x877af9 (0x1baa76 bytes) 000474[1] 1769073649.469223 mapwrite 0x9db000 thru 0x9deb33 (0x3b34 bytes) 000475[1] 1769073649.753942 write_OD 0x3d5000 thru 0x3ddfff (0x9000 bytes) 000477[1] 1769073649.824161 mapwrite 0x503000 thru 0x50c58a (0x958b bytes) 000482[0] 1769073650.020631 read 0x0f7000 thru 0x0fa1a6 (0x31a7 bytes) 000484[0] 1769073650.046183 mapread 0x1ec000 thru 0x1efdb9 (0x3dba bytes) 000487[0] 1769073650.069049 trunc from 0x9deb34 to 0x7a0103 (0x23ea30 bytes) 000489[1] 1769073650.104966 mapwrite 0x4e2000 thru 0x4e91df (0x71e0 bytes) 000490[0] 1769073650.228173 write 0x26e000 thru 0x274a87 (0x6a88 bytes) 000492[1] 1769073650.249583 trunc from 0x7a0104 to 0x0d3520 (0x6ccbe3 bytes) 000493[0] 1769073650.291211 trunc from 0x0d3521 to 0x46405e (0x390b3e bytes) 000495[0] 1769073650.349503 mapwrite 0x2eb000 thru 0x2f24da (0x74db bytes) 000499[0] 1769073650.456813 mapwrite 0x222000 thru 0x22c90e (0xa90f bytes) All operations completed A-OK! PASS 16a (60s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:20:56 (1769073656) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.447231 s, 18.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.330228 s, 25.4 MB/s Chance of close/open is 1 in 50 Seed set to 3659 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1769073660.191223 mapwrite 0x623363 thru 0x624b66 (0x1804 bytes) 000004[1] 1769073660.322924 mapread 0x3a683e thru 0x3a8545 (0x1d08 bytes) 000005[1] 1769073660.429358 mapread 0x265708 thru 0x265e50 (0x749 bytes) 000006[0] 1769073660.464243 read 0x3b6b81 thru 0x3b81b8 (0x1638 bytes) 000008[1] 1769073660.509719 mapread 0x074f86 thru 0x076363 (0x13de bytes) 000009[0] 1769073660.545730 write 0x3e14ee thru 0x3e246f (0xf82 bytes) truncating to largest ever: 0x788ef 000011[1] 1769073660.632377 trunc from 0x624b67 to 0x0788ee (0x5ac278 bytes) truncating to largest ever: 0x5b4f8c 000012[0] 1769073660.806829 trunc from 0x0788ef to 0x5b4f8b (0x53c69d bytes) 000013[1] 1769073660.927428 write 0x89eca0 thru 0x89ed5f (0xc0 bytes) 000016[0] 1769073661.054104 read 0x6694ad thru 0x66a1b6 (0xd0a bytes) 000021[1] 1769073661.281792 read 0x7136b0 thru 0x71449a (0xdeb bytes) 000023[0] 1769073661.307163 write 0x763aaf thru 0x763ae7 (0x39 bytes) 000024[0] 1769073661.366607 read 0x04e036 thru 0x04fd3e (0x1d09 bytes) 000026[1] 1769073661.437152 mapread 0x411006 thru 0x412832 (0x182d bytes) 000028[1] 1769073661.470417 trunc from 0x89ed60 to 0x007e2e (0x896f31 bytes) 000032[1] 1769073661.542632 mapread 0x006aa9 thru 0x007e2e (0x1386 bytes) 000034[0] 1769073661.573980 mapread 0x002a9f thru 0x003282 (0x7e4 bytes) 000035[1] 1769073661.623263 mapread 0x007345 thru 0x0073f0 (0xac bytes) truncating to largest ever: 0x5fc1dc 000036[1] 1769073661.648884 trunc from 0x007e2f to 0x5fc1db (0x5f43ad bytes) 000037[1] 1769073661.712360 read 0x292b7e thru 0x29447d (0x1900 bytes) 000040[0] 1769073661.753019 mapwrite 0x47efaa thru 0x480707 (0x175e bytes) 000042[0] 1769073661.873306 read 0x467a1c thru 0x468de7 (0x13cc bytes) 000043[0] 1769073661.916230 mapread 0x182ee4 thru 0x183dfb (0xf18 bytes) 000044[0] 1769073662.012485 mapwrite 0x2d34dd thru 0x2d547b (0x1f9f bytes) 000047[1] 1769073662.309364 write 0x67f664 thru 0x681068 (0x1a05 bytes) 000048[1] 1769073662.369199 trunc from 0x681069 to 0x3123a0 (0x36ecc8 bytes) 000050[1] 1769073662.536616 trunc from 0x3123a1 to 0x2f4fda (0x1d3c6 bytes) 000051[1] 1769073662.558007 trunc from 0x2f4fdb to 0x106f84 (0x1ee056 bytes) 000054[1] 1769073662.631039 mapwrite 0x70ffbe thru 0x711256 (0x1299 bytes) 000055[0] 1769073662.799688 write 0x74cde7 thru 0x74d6ff (0x919 bytes) 000056[0] 1769073662.842063 write 0x928541 thru 0x928a00 (0x4c0 bytes) truncating to largest ever: 0x7ae2cb 000060[0] 1769073662.926046 trunc from 0x928a01 to 0x7ae2ca (0x17a736 bytes) 000063[1] 1769073662.977666 mapread 0x26a9c3 thru 0x26ba23 (0x1061 bytes) 000064[0] 1769073663.031146 mapwrite 0x2f8b40 thru 0x2f902c (0x4ed bytes) 000065[1] 1769073663.121081 read 0x1ee3d6 thru 0x1ef7c2 (0x13ed bytes) 000066[1] 1769073663.152966 trunc from 0x7ae2cb to 0x305a25 (0x4a88a5 bytes) 000069[0] 1769073663.226739 write 0x867f66 thru 0x869a7e (0x1b19 bytes) 000071[1] 1769073663.291270 read 0x55de69 thru 0x55fe46 (0x1fde bytes) 000073[1] 1769073663.352265 write 0x71c48a thru 0x71d1db (0xd52 bytes) 000074[0] 1769073663.374815 read 0x06fe39 thru 0x0717ce (0x1996 bytes) 000075[0] 1769073663.410402 mapwrite 0x4e1c54 thru 0x4e309a (0x1447 bytes) 000078[0] 1769073663.554301 read 0x1620a0 thru 0x162d9e (0xcff bytes) 000079[1] 1769073663.588184 trunc from 0x869a7f to 0x5c6ac0 (0x2a2fbe bytes) 000080[1] 1769073663.660011 mapread 0x05b777 thru 0x05b7e1 (0x6b bytes) 000083[1] 1769073663.712977 mapwrite 0x489127 thru 0x4894be (0x398 bytes) 000084[0] 1769073663.813360 mapwrite 0x83c7a3 thru 0x83d925 (0x1183 bytes) 000085[1] 1769073663.924938 read 0x4b8634 thru 0x4b9fdc (0x19a9 bytes) 000086[1] 1769073663.958341 mapread 0x6ce7de thru 0x6cebc7 (0x3ea bytes) 000088[1] 1769073663.974026 trunc from 0x83d926 to 0x35d8cd (0x4e0058 bytes) 000089[1] 1769073664.034856 mapread 0x3227b7 thru 0x324142 (0x198c bytes) 000095[0] 1769073664.180004 read 0x2991bc thru 0x29a745 (0x158a bytes) 000097[1] 1769073664.298261 trunc from 0x35d8ce to 0x18cb96 (0x1d0d37 bytes) 000098[0] 1769073664.361846 mapwrite 0x0113fd thru 0x012880 (0x1484 bytes) 000101[1] 1769073664.465524 trunc from 0x18cb97 to 0x10a36e (0x82828 bytes) 000102[1] 1769073664.539203 read 0x000fac thru 0x002a9b (0x1af0 bytes) 000104[0] 1769073664.585257 write 0x9940b9 thru 0x995c0b (0x1b53 bytes) 000106[1] 1769073664.687298 mapread 0x967825 thru 0x968426 (0xc02 bytes) 000107[0] 1769073664.919425 read 0x4f5a18 thru 0x4f60dd (0x6c6 bytes) 000108[1] 1769073664.997168 mapwrite 0x1d92fd thru 0x1dac78 (0x197c bytes) 000110[1] 1769073665.132414 read 0x00ca7b thru 0x00d3f5 (0x97b bytes) 000113[1] 1769073665.238946 mapwrite 0x243bf1 thru 0x244ba1 (0xfb1 bytes) 000115[0] 1769073665.356249 read 0x02795b thru 0x02922b (0x18d1 bytes) 000116[0] 1769073665.388315 mapwrite 0x2c3ef0 thru 0x2c4778 (0x889 bytes) 000118[1] 1769073665.561165 mapread 0x7be7ce thru 0x7bf19c (0x9cf bytes) 000119[1] 1769073665.635256 mapwrite 0x4f0da6 thru 0x4f2aea (0x1d45 bytes) 000121[0] 1769073665.945513 write 0x444d55 thru 0x444f91 (0x23d bytes) 000124[0] 1769073666.029334 mapwrite 0x3a4d78 thru 0x3a5b86 (0xe0f bytes) 000125[1] 1769073666.209721 mapread 0x4815ac thru 0x482f5e (0x19b3 bytes) 000126[0] 1769073666.303337 mapread 0x6fe53b thru 0x6fe929 (0x3ef bytes) 000128[0] 1769073666.346085 read 0x067cb9 thru 0x067d20 (0x68 bytes) 000129[0] 1769073666.401469 read 0x8ac8f2 thru 0x8acba8 (0x2b7 bytes) 000130[1] 1769073666.409148 mapread 0x3e5ba6 thru 0x3e7978 (0x1dd3 bytes) 000131[0] 1769073666.512112 write 0x2f908d thru 0x2fabad (0x1b21 bytes) 000136[1] 1769073666.622252 mapwrite 0x48e4b8 thru 0x48e5e4 (0x12d bytes) 000140[1] 1769073666.874182 read 0x8fedb1 thru 0x900c2a (0x1e7a bytes) 000144[1] 1769073666.917353 trunc from 0x995c0c to 0x648328 (0x34d8e3 bytes) 000145[1] 1769073666.992294 trunc from 0x648329 to 0x623fba (0x2436e bytes) 000146[0] 1769073667.015810 mapwrite 0x2b17ad thru 0x2b1978 (0x1cc bytes) 000148[1] 1769073667.123267 write 0x804a84 thru 0x804b53 (0xd0 bytes) 000151[0] 1769073667.169908 write 0x931b80 thru 0x933633 (0x1ab4 bytes) 000152[0] 1769073667.213080 read 0x362133 thru 0x3635ac (0x147a bytes) 000153[1] 1769073667.261770 mapwrite 0x59d894 thru 0x59eed3 (0x1640 bytes) 000154[1] 1769073667.387332 mapwrite 0x93ae6f thru 0x93b2bd (0x44f bytes) 000157[1] 1769073667.655901 mapwrite 0x0d0398 thru 0x0d0f95 (0xbfe bytes) 000162[0] 1769073667.903010 mapwrite 0x9135c7 thru 0x914e91 (0x18cb bytes) 000164[1] 1769073668.197303 write 0x2955cb thru 0x29639c (0xdd2 bytes) 000165[1] 1769073668.246433 read 0x47315e thru 0x474e96 (0x1d39 bytes) 000167[1] 1769073668.316586 write 0x71a7cc thru 0x71b3d6 (0xc0b bytes) 000168[0] 1769073668.386031 trunc from 0x93b2be to 0x47c6b4 (0x4bec09 bytes) 000174[1] 1769073668.714296 write 0x4a4438 thru 0x4a4544 (0x10d bytes) 000176[1] 1769073668.853819 write 0x765884 thru 0x766c1b (0x1398 bytes) 000178[0] 1769073668.923950 write 0x96184c thru 0x962cb4 (0x1469 bytes) 000180[0] 1769073668.956735 read 0x4a24a4 thru 0x4a2f36 (0xa93 bytes) 000181[0] 1769073668.998213 read 0x4d7e35 thru 0x4d8003 (0x1cf bytes) 000183[0] 1769073669.019746 mapwrite 0x8e2c10 thru 0x8e465f (0x1a50 bytes) 000187[0] 1769073669.132196 trunc from 0x962cb5 to 0x352dc9 (0x60feeb bytes) 000188[0] 1769073669.233938 trunc from 0x352dca to 0x6ddf14 (0x38b14b bytes) 000189[0] 1769073669.281007 read 0x416164 thru 0x417fa4 (0x1e41 bytes) 000190[1] 1769073669.340258 mapwrite 0x9fdb73 thru 0x9fdbea (0x78 bytes) 000192[1] 1769073669.575416 write 0x9440ba thru 0x945ed5 (0x1e1c bytes) 000194[1] 1769073669.625936 mapwrite 0x18e8da thru 0x18ef7b (0x6a2 bytes) 000195[0] 1769073669.752723 trunc from 0x9fdbeb to 0x767d07 (0x295ee3 bytes) 000200[1] 1769073669.921607 mapread 0x4887ec thru 0x488e3f (0x654 bytes) 000202[1] 1769073670.048813 mapwrite 0x8161aa thru 0x816a72 (0x8c9 bytes) 000205[0] 1769073670.267675 read 0x3eecf7 thru 0x3f0cab (0x1fb5 bytes) 000206[0] 1769073670.307517 mapread 0x5eecff thru 0x5efbf9 (0xefb bytes) 000208[0] 1769073670.337822 read 0x6104fd thru 0x610e77 (0x97b bytes) 000211[1] 1769073670.344290 mapwrite 0x083ab3 thru 0x0840d2 (0x620 bytes) 000217[0] 1769073670.639508 mapwrite 0x7651fa thru 0x765e25 (0xc2c bytes) 000219[1] 1769073670.908206 mapread 0x272c0a thru 0x272e82 (0x279 bytes) 000221[1] 1769073671.054137 write 0x68f82d thru 0x6900bf (0x893 bytes) 000222[0] 1769073671.160866 write 0x52f3a8 thru 0x530ea2 (0x1afb bytes) 000223[0] 1769073671.287671 mapwrite 0x8a7c91 thru 0x8a9239 (0x15a9 bytes) 000224[1] 1769073671.425829 trunc from 0x8a923a to 0x7523a6 (0x156e93 bytes) 000225[1] 1769073671.552996 write 0x75ff9c thru 0x7613cc (0x1431 bytes) 000226[1] 1769073671.563212 write 0x0b75f9 thru 0x0b83cb (0xdd3 bytes) 000228[1] 1769073671.620178 trunc from 0x7613cd to 0x058821 (0x708bab bytes) 000229[0] 1769073671.657703 trunc from 0x058822 to 0x76809e (0x70f87d bytes) 000235[0] 1769073671.731752 trunc from 0x76809f to 0x66495f (0x10373f bytes) 000236[0] 1769073671.752903 mapwrite 0x15c5b9 thru 0x15df6e (0x19b6 bytes) 000238[1] 1769073671.896849 mapread 0x388bff thru 0x3895f1 (0x9f3 bytes) 000239[1] 1769073671.964983 mapread 0x37cf6b thru 0x37dbcf (0xc65 bytes) 000241[1] 1769073672.058095 read 0x021cbd thru 0x022bf3 (0xf37 bytes) 000242[0] 1769073672.075767 read 0x3051f1 thru 0x305307 (0x117 bytes) 000243[0] 1769073672.087257 mapwrite 0x965122 thru 0x96561d (0x4fc bytes) 000245[0] 1769073672.510136 mapwrite 0x88fdb3 thru 0x8916fc (0x194a bytes) 000246[0] 1769073672.671365 read 0x60ff5f thru 0x610228 (0x2ca bytes) truncating to largest ever: 0x852403 000248[1] 1769073672.726819 trunc from 0x96561e to 0x852402 (0x11321b bytes) 000250[1] 1769073672.829848 mapwrite 0x5736c7 thru 0x575063 (0x199d bytes) 000252[0] 1769073672.902465 mapread 0x4735df thru 0x4751af (0x1bd1 bytes) 000254[1] 1769073673.125744 write 0x0e5f9c thru 0x0e7be1 (0x1c46 bytes) 000256[0] 1769073673.222103 mapwrite 0x98886d thru 0x989ba8 (0x133c bytes) 000260[0] 1769073673.552909 mapread 0x87fea1 thru 0x8812ef (0x144f bytes) 000261[0] 1769073673.594047 write 0x0c2c57 thru 0x0c3213 (0x5bd bytes) 000262[0] 1769073673.606482 write 0x77be58 thru 0x77c5d4 (0x77d bytes) 000264[1] 1769073673.636622 mapread 0x6c4023 thru 0x6c4023 (0x1 bytes) 000268[0] 1769073673.676281 write 0x98a6d6 thru 0x98bd15 (0x1640 bytes) 000271[1] 1769073673.695991 write 0x959ef8 thru 0x95a25f (0x368 bytes) 000272[0] 1769073673.776724 trunc from 0x98bd16 to 0x3b7179 (0x5d4b9c bytes) 000273[1] 1769073673.861293 trunc from 0x3b717a to 0x6f5cb3 (0x33eb3a bytes) 000274[1] 1769073673.929437 mapwrite 0x1bc57d thru 0x1bcd52 (0x7d6 bytes) 000280[0] 1769073674.065758 write 0x33d6bb thru 0x33eee5 (0x182b bytes) 000281[1] 1769073674.099847 read 0x482d67 thru 0x4837d1 (0xa6b bytes) 000284[1] 1769073674.152334 write 0x0cedb3 thru 0x0d08ad (0x1afb bytes) 000285[1] 1769073674.180852 read 0x163a0f thru 0x164cba (0x12ac bytes) 000286[1] 1769073674.206789 read 0x3091bd thru 0x3098db (0x71f bytes) 000288[0] 1769073674.238273 mapwrite 0x8471db thru 0x848d40 (0x1b66 bytes) 000290[1] 1769073674.389835 read 0x04f6e0 thru 0x050f9d (0x18be bytes) 000291[0] 1769073674.432238 write 0x17660f thru 0x176829 (0x21b bytes) 000292[1] 1769073674.482193 read 0x46c599 thru 0x46e41d (0x1e85 bytes) 000296[1] 1769073674.544145 trunc from 0x848d41 to 0x333bd4 (0x51516c bytes) 000300[1] 1769073674.731079 trunc from 0x333bd5 to 0x40ad89 (0xd71b5 bytes) 000301[0] 1769073674.760214 write 0x71489d thru 0x71636e (0x1ad2 bytes) 000302[1] 1769073674.801618 write 0x6ba5cc thru 0x6bb5f4 (0x1029 bytes) 000303[0] 1769073674.831895 mapwrite 0x98a92f thru 0x98ab49 (0x21b bytes) 000304[1] 1769073674.982103 mapread 0x62cadd thru 0x62d8c9 (0xded bytes) 000305[1] 1769073675.054922 write 0x43a95d thru 0x43c21b (0x18bf bytes) 000306[0] 1769073675.113101 trunc from 0x98ab4a to 0x017027 (0x973b22 bytes) 000311[1] 1769073675.224383 mapread 0x004cee thru 0x006385 (0x1698 bytes) 000315[0] 1769073675.398906 read 0x0105d2 thru 0x010d8a (0x7b9 bytes) 000316[0] 1769073675.419760 mapread 0x003fe5 thru 0x00505e (0x107a bytes) 000317[0] 1769073675.449998 mapwrite 0x1665db thru 0x1675e0 (0x1006 bytes) 000318[0] 1769073675.553188 mapread 0x0f2ff3 thru 0x0f38a5 (0x8b3 bytes) 000320[0] 1769073675.615866 read 0x00757c thru 0x008413 (0xe98 bytes) 000321[1] 1769073675.670253 trunc from 0x1675e1 to 0x46e3f2 (0x306e12 bytes) 000323[0] 1769073675.845913 mapwrite 0x2961e0 thru 0x2968e9 (0x70a bytes) 000324[0] 1769073676.233670 mapread 0x22956c thru 0x22a6c9 (0x115e bytes) 000325[1] 1769073676.303802 trunc from 0x46e3f3 to 0x2af39b (0x1bf057 bytes) 000326[0] 1769073676.378075 read 0x1b7e65 thru 0x1b8756 (0x8f2 bytes) 000327[0] 1769073676.445183 mapread 0x1e7fb0 thru 0x1e8617 (0x668 bytes) 000329[1] 1769073676.588236 mapwrite 0x1de8da thru 0x1dfb19 (0x1240 bytes) 000331[1] 1769073676.910375 trunc from 0x2af39c to 0x008891 (0x2a6b0a bytes) 000332[1] 1769073676.934576 read 0x003a89 thru 0x0047a6 (0xd1e bytes) 000333[1] 1769073677.021357 mapread 0x0030dd thru 0x004c1b (0x1b3f bytes) 000334[0] 1769073677.023351 read 0x003580 thru 0x0051a2 (0x1c23 bytes) 000336[0] 1769073677.115898 mapread 0x005fd1 thru 0x0065ea (0x61a bytes) 000338[1] 1769073677.161257 mapwrite 0x2524cb thru 0x2536a7 (0x11dd bytes) 000342[1] 1769073677.414845 trunc from 0x2536a8 to 0x7cf9d2 (0x57c32b bytes) 000343[0] 1769073677.483006 write 0x2336c7 thru 0x23478e (0x10c8 bytes) 000344[0] 1769073677.651699 mapwrite 0x9f4851 thru 0x9f615c (0x190c bytes) 000345[0] 1769073677.874380 write 0x29f12c thru 0x2a107e (0x1f53 bytes) 000346[0] 1769073677.913891 read 0x8a5d6b thru 0x8a657c (0x812 bytes) 000347[0] 1769073677.939021 mapwrite 0x27b818 thru 0x27be8d (0x676 bytes) 000348[1] 1769073678.051625 trunc from 0x9f615d to 0x1fb5e0 (0x7fab7c bytes) 000349[0] 1769073678.191603 read 0x0fb932 thru 0x0fc36f (0xa3e bytes) 000350[1] 1769073678.239822 mapread 0x01915a thru 0x019e05 (0xcac bytes) 000351[1] 1769073678.287161 write 0x3a0647 thru 0x3a19e0 (0x139a bytes) 000353[0] 1769073678.366083 read 0x2e563d thru 0x2e7298 (0x1c5c bytes) 000354[1] 1769073678.543447 write 0x4b64e3 thru 0x4b809a (0x1bb8 bytes) 000355[1] 1769073678.590922 mapwrite 0x2c2bbb thru 0x2c300c (0x452 bytes) 000357[1] 1769073678.869245 trunc from 0x4b809b to 0x27a36d (0x23dd2d bytes) 000358[1] 1769073678.933349 read 0x1728de thru 0x172d85 (0x4a8 bytes) 000359[1] 1769073679.143105 write 0x7b31b5 thru 0x7b4dda (0x1c26 bytes) 000363[1] 1769073679.211936 read 0x69ab2e thru 0x69c645 (0x1b18 bytes) 000364[1] 1769073679.272403 write 0x83f31d thru 0x8408de (0x15c2 bytes) 000366[1] 1769073679.364233 mapwrite 0x279ef8 thru 0x27a890 (0x999 bytes) 000367[1] 1769073679.512708 write 0x679274 thru 0x67b079 (0x1e06 bytes) 000368[0] 1769073679.575056 mapwrite 0x2e1e5e thru 0x2e3cd5 (0x1e78 bytes) 000370[0] 1769073679.890830 mapwrite 0x0a5488 thru 0x0a5917 (0x490 bytes) 000371[1] 1769073680.049016 mapread 0x75b185 thru 0x75b38c (0x208 bytes) 000373[0] 1769073680.178978 read 0x5c4c77 thru 0x5c5fe0 (0x136a bytes) 000376[1] 1769073680.326317 mapread 0x5bb860 thru 0x5bd66b (0x1e0c bytes) 000377[1] 1769073680.424233 mapread 0x002280 thru 0x003162 (0xee3 bytes) 000379[0] 1769073680.535748 trunc from 0x8408df to 0x5afa0e (0x290ed0 bytes) 000382[0] 1769073680.728052 mapread 0x482bf3 thru 0x482c56 (0x64 bytes) 000384[0] 1769073680.815845 mapread 0x0ba56b thru 0x0bbfe9 (0x1a7f bytes) 000386[1] 1769073680.847727 mapwrite 0x5591f1 thru 0x55ae71 (0x1c81 bytes) 000388[0] 1769073681.098208 read 0x2b2737 thru 0x2b38cd (0x1197 bytes) 000390[1] 1769073681.154319 trunc from 0x5afa0f to 0x3dfaf7 (0x1cff17 bytes) 000392[1] 1769073681.317841 write 0x95e3f5 thru 0x95fd20 (0x192c bytes) 000395[1] 1769073681.337969 write 0x94b53b thru 0x94c5bc (0x1082 bytes) 000397[1] 1769073681.456929 mapwrite 0x435153 thru 0x435c67 (0xb15 bytes) 000398[1] 1769073681.613820 mapwrite 0x91ef25 thru 0x920556 (0x1632 bytes) 000399[1] 1769073681.961299 trunc from 0x95fd21 to 0x2a618e (0x6b9b92 bytes) 000400[1] 1769073681.999003 mapread 0x281990 thru 0x283103 (0x1774 bytes) 000401[1] 1769073682.137074 read 0x1986e4 thru 0x199f82 (0x189f bytes) 000402[1] 1769073682.313080 mapwrite 0x820191 thru 0x821eca (0x1d3a bytes) 000404[0] 1769073682.675383 mapread 0x11c6c6 thru 0x11c76f (0xaa bytes) 000406[0] 1769073682.800258 write 0x53143a thru 0x532097 (0xc5e bytes) 000408[1] 1769073682.866786 read 0x2624fb thru 0x263f92 (0x1a98 bytes) 000409[0] 1769073682.875750 read 0x279c1c thru 0x27ac3d (0x1022 bytes) 000410[1] 1769073682.884731 mapwrite 0x529cce thru 0x529f0b (0x23e bytes) 000411[0] 1769073683.203728 read 0x1ff055 thru 0x200beb (0x1b97 bytes) 000412[1] 1769073683.221849 mapwrite 0x69a26d thru 0x69b0f1 (0xe85 bytes) 000413[0] 1769073683.382487 write 0x56b680 thru 0x56d44b (0x1dcc bytes) 000414[0] 1769073683.487456 mapwrite 0x88bc45 thru 0x88c820 (0xbdc bytes) 000415[1] 1769073683.711856 write 0x3b8bc7 thru 0x3b8f38 (0x372 bytes) 000416[0] 1769073683.781658 write 0x02a133 thru 0x02b611 (0x14df bytes) 000417[0] 1769073683.873956 mapwrite 0x8c82e2 thru 0x8ca007 (0x1d26 bytes) 000418[1] 1769073684.101971 trunc from 0x8ca008 to 0x283b75 (0x646492 bytes) 000422[0] 1769073684.370431 mapwrite 0x5f6d26 thru 0x5f827d (0x1558 bytes) 000424[0] 1769073684.667644 read 0x0a54ce thru 0x0a5f0d (0xa40 bytes) 000428[0] 1769073684.770913 read 0x21307e thru 0x213fde (0xf61 bytes) 000432[0] 1769073684.781290 mapread 0x3ffbe6 thru 0x4017dc (0x1bf7 bytes) 000433[1] 1769073684.817465 read 0x00340b thru 0x00533e (0x1f34 bytes) 000435[0] 1769073684.881222 mapwrite 0x7aba63 thru 0x7acc63 (0x1201 bytes) skipping zero size read 000437[0] 1769073685.075352 write 0x621b24 thru 0x6239b1 (0x1e8e bytes) 000438[1] 1769073685.104084 mapwrite 0x039fe9 thru 0x03abfb (0xc13 bytes) 000439[0] 1769073685.204572 read 0x18f9db thru 0x1900a3 (0x6c9 bytes) 000440[1] 1769073685.232909 mapwrite 0x858ad1 thru 0x859096 (0x5c6 bytes) 000444[1] 1769073685.397517 mapread 0x5ce4a7 thru 0x5ceb3b (0x695 bytes) 000445[1] 1769073685.403793 mapread 0x04d18c thru 0x04d351 (0x1c6 bytes) 000446[1] 1769073685.439407 mapread 0x2c4699 thru 0x2c5852 (0x11ba bytes) 000447[1] 1769073685.455157 read 0x6db933 thru 0x6dd184 (0x1852 bytes) 000448[0] 1769073685.463634 mapread 0x039a94 thru 0x03b557 (0x1ac4 bytes) 000449[0] 1769073685.550158 trunc from 0x859097 to 0x83bee9 (0x1d1ad bytes) truncating to largest ever: 0x955c1c 000451[0] 1769073685.714657 trunc from 0x83beea to 0x955c1b (0x119d32 bytes) 000453[0] 1769073685.800807 write 0x8b4f74 thru 0x8b624c (0x12d9 bytes) 000455[1] 1769073685.862442 mapread 0x5ce34b thru 0x5ce770 (0x426 bytes) 000457[1] 1769073685.940231 mapwrite 0x9fac30 thru 0x9fbbf9 (0xfca bytes) 000458[1] 1769073686.146426 read 0x5a7a35 thru 0x5a9976 (0x1f42 bytes) 000460[1] 1769073686.170491 mapread 0x35eb32 thru 0x35fcf6 (0x11c5 bytes) 000461[1] 1769073686.191476 read 0x591213 thru 0x5930a6 (0x1e94 bytes) 000462[0] 1769073686.238963 write 0x15a315 thru 0x15abb9 (0x8a5 bytes) 000463[0] 1769073686.322636 read 0x6555dc thru 0x65693f (0x1364 bytes) 000465[0] 1769073686.385191 write 0x1a9d31 thru 0x1aaaf6 (0xdc6 bytes) 000466[0] 1769073686.430529 mapwrite 0x2e53fa thru 0x2e5c46 (0x84d bytes) 000470[1] 1769073686.525487 mapwrite 0x655e1e thru 0x657537 (0x171a bytes) 000471[1] 1769073686.639033 write 0x86fff0 thru 0x8715df (0x15f0 bytes) 000472[0] 1769073686.700017 trunc from 0x9fbbfa to 0x582710 (0x4794e9 bytes) 000473[1] 1769073686.830963 write 0x8e8f37 thru 0x8eade4 (0x1eae bytes) 000474[0] 1769073686.899817 read 0x1d6a12 thru 0x1d872d (0x1d1c bytes) 000476[1] 1769073687.032994 read 0x31c13f thru 0x31d74b (0x160d bytes) 000479[0] 1769073687.059319 write 0x7c5f65 thru 0x7c7cd4 (0x1d70 bytes) 000481[1] 1769073687.098246 trunc from 0x8eade5 to 0x845f37 (0xa4ead bytes) 000483[0] 1769073687.216321 write 0x0620c6 thru 0x064059 (0x1f94 bytes) 000488[1] 1769073687.323423 mapwrite 0x2c7115 thru 0x2c775b (0x647 bytes) 000492[0] 1769073687.445850 read 0x5097ca thru 0x50ac5a (0x1491 bytes) 000495[0] 1769073687.471673 mapread 0x5351be thru 0x536418 (0x125b bytes) 000496[0] 1769073687.490185 write 0x1029dd thru 0x103929 (0xf4d bytes) All operations completed A-OK! PASS 16b (37s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:21:33 (1769073693) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:21:38 (1769073698) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.432294 s, 19.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.343692 s, 24.4 MB/s ldlm.namespaces.MGC192.168.203.145@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8bd6c9830000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8bd6d5282800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8bd6c9830000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8bd6c9830000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.9445 s, 6.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.35429 s, 24.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.04073 s, 34.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.44636 s, 16.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.49106 s, 11.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.20581 s, 24.9 MB/s PASS 16d (58s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:22:36 (1769073756) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260812 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194592 s, 5.4 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:22:44 (1769073764) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:23:13 (1769073793) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 04:23:41 (1769073821) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 04:23:49 (1769073829) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 04:23:57 (1769073837) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17818 s, 8.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.92997 s, 13.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.9847 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22056 s, 11.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8248 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.40027 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.81907 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33984 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45349 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.61534 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.71892 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33724 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48951 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04078 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.23474 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.71163 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95719 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60269 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.65005 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (47s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:24:44 (1769073884) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Jan 22 04:25:44 EST 2026 + fsx 2 done: rc=0 Thu Jan 22 04:25:44 EST 2026 + fsx 3 done: rc=0 Thu Jan 22 04:25:44 EST 2026 + fsx 4 done: rc=0 Thu Jan 22 04:25:44 EST 2026 + fsx 5 done: rc=0 Thu Jan 22 04:25:44 EST 2026 PASS 16k (74s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:25:58 (1769073958) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 04:26:12 (1769073972) mmap test1: basic mmap operation (PASS, 0.18s) mmap test2: MAP_PRIVATE not write back (PASS, 0.085641s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.6903s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6772s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.27285s) mmap test6: check mmap write/read content on two nodes (PASS, 0.38801s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3269s) mmap test11: multi-thread shared mmap (PASS, 1.4784s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1624s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:26:39 (1769073999) oleg345-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:26:44 (1769074004) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:26:53 (1769074013) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 04:27:00 (1769074020) now is 1769074023 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 new atime is 1769074084 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:28:10 (1769074090) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:28:19 (1769074099) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:28:27 (1769074107) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:28:37 (1769074117) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 04:28:41 (1769074121) utime: good mknod times 1769074122 <= 1769074123 <= 1769074123 for /mnt/lustre/f26a utime: good mknod times 1769074122 <= 1769074123 <= 1769074123 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 04:28:49 (1769074129) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 04:28:59 (1769074139) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-22 04:29:01.000000000 -0500 Birth: 2026-01-22 04:29:00.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-22 04:29:01.000000000 -0500 Birth: 2026-01-22 04:29:00.000000000 -0500 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 04:29:11 (1769074151) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:29:20 (1769074160) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:29:28 (1769074168) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.145@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 04:29:51 (1769074191) -rw-r--r-- 1 root root 0 Jan 22 04:29 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 04:30:02 (1769074202) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42896 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 04:30:10 (1769074210) PASS 31t (8s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:30:20 (1769074220) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 04:30:23 (1769074223) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 04:30:26 (1769074226) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 04:30:30 (1769074230) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 04:30:34 (1769074234) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid in IDLE state after 0 sec oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid in FULL state after 0 sec oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid in IDLE state after 0 sec oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid in FULL state after 0 sec oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd6c9830000.ost_server_uuid in IDLE state after 0 sec oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd6c9830000.ost_server_uuid in FULL state after 0 sec PASS 34 (130s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 04:32:44 (1769074364) fail_loc=0x80000317 Race attempt 0 Wait for 48957 48995 for 60 sec... - open/close 685 (time 1769074381.40 total 10.01 last 68.43) - open/close 1415 (time 1769074391.41 total 20.02 last 72.96) - open/close 2223 (time 1769074401.41 total 30.02 last 80.80) - open/close 2940 (time 1769074411.41 total 40.02 last 71.64) - open/close 3715 (time 1769074421.42 total 50.02 last 77.49) total: 4000 open/close in 53.42 seconds: 74.87 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.145@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.145@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 04:33:56 (1769074436) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.79126 s, 29.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41965 s, 36.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.55491 s, 33.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30158 s, 40.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47033 s, 35.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7489536) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57282 s, 33.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64398 s, 31.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53213 s, 34.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64516 s, 31.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46966 s, 35.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43245 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7443456) after(7494656) PASS 36 (298s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 04:38:54 (1769074734) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 - create 1337 (time 1769074748.39 total 10.01 last 133.60) - create 2582 (time 1769074758.39 total 20.01 last 124.47) - create 3817 (time 1769074768.40 total 30.01 last 123.45) - create 5169 (time 1769074778.40 total 40.02 last 135.15) - create 6174 (time 1769074788.41 total 50.03 last 100.39) - create 7165 (time 1769074798.41 total 60.03 last 99.09) - create 8396 (time 1769074808.41 total 70.03 last 123.10) - create 9929 (time 1769074818.41 total 80.03 last 153.25) total: 10000 create in 80.66 seconds: 123.98 ops/second PASS 37 (92s) == sanityn test 39a: file mtime does not change after rename ========================================================== 04:40:26 (1769074826) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 04:40:34 (1769074834) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 04:40:42 (1769074842) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 04:40:49 (1769074849) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 04:40:58 (1769074858) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 04:41:16 (1769074876) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 04:41:33 (1769074893) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 04:41:50 (1769074910) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 04:42:08 (1769074928) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 04:42:21 (1769074941) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 04:42:34 (1769074954) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 04:42:47 (1769074967) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 04:43:00 (1769074980) Conflict PASS 41e (16s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 04:43:15 (1769074995) Conflict PASS 41f (16s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 04:43:31 (1769075011) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 04:43:46 (1769075026) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (17s) == sanityn test 41i: reint_open: create vs create ======== 04:44:03 (1769075043) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1088s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 05:02:11 (1769076131) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 05:02:26 (1769076146) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 05:02:41 (1769076161) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 05:02:54 (1769076174) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 05:03:09 (1769076189) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 05:03:25 (1769076205) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 05:03:41 (1769076221) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 05:03:55 (1769076235) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 05:04:08 (1769076248) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 05:05:29 (1769076329) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 05:05:43 (1769076343) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 05:05:58 (1769076358) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 05:06:12 (1769076372) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 05:06:28 (1769076388) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 05:06:42 (1769076402) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 05:06:58 (1769076418) Conflict PASS 43h (20s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 05:07:18 (1769076438) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 05:07:21 (1769076441) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (143s) == sanityn test 43k: unlink vs create ==================== 05:09:44 (1769076584) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1333s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 05:31:57 (1769077917) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Jan 22 05:32:05 EST 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 05:32:10 (1769077930) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 05:32:23 (1769077943) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 05:32:36 (1769077956) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 05:32:49 (1769077969) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 05:33:02 (1769077982) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 05:33:17 (1769077997) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 05:33:32 (1769078012) Conflict PASS 44h (16s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 05:33:49 (1769078029) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 05:33:53 (1769078033) PASS 45a (132s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 05:36:06 (1769078166) Conflict PASS 45b (24s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 05:36:30 (1769078190) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 05:36:46 (1769078206) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 05:37:02 (1769078222) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 05:37:16 (1769078236) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (16s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 05:37:32 (1769078252) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 05:37:47 (1769078267) Conflict PASS 45h (16s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 05:38:03 (1769078283) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 05:38:08 (1769078288) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1224s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 05:58:32 (1769079512) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 05:58:44 (1769079524) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 05:58:55 (1769079535) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (11s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 05:59:06 (1769079546) Conflict PASS 46d (10s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 05:59:16 (1769079556) Conflict PASS 46e (10s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 05:59:26 (1769079566) Conflict PASS 46f (11s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 05:59:37 (1769079577) Conflict PASS 46g (11s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 05:59:48 (1769079588) Conflict PASS 46h (11s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 05:59:59 (1769079599) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:00:02 (1769079602) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:00:04 (1769079604) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:00:06 (1769079606) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:00:09 (1769079609) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:00:11 (1769079611) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:00:13 (1769079613) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:00:16 (1769079616) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:00:18 (1769079618) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0363499 s, 282 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (10s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:00:28 (1769079628) 0+1 records in 0+1 records out 158 bytes copied, 0.0197836 s, 8.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (215994) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:00:35 (1769079635) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00670111 s, 153 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663879 s, 15.8 MB/s 3145728 PASS 51b (17s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:00:52 (1769079652) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0203979 s, 50.2 kB/s PASS 51c (10s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:01:02 (1769079662) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250568 s, 41.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (6s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:01:08 (1769079668) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 06:01:14 (1769079674) llite.lustre-ffff8bd6c9830000.xattr_cache=1 llite.lustre-ffff8bd6d5282800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:15.000000000 -0500 Modify: 2026-01-22 06:01:15.000000000 -0500 Change: 2026-01-22 06:01:15.000000000 -0500 Birth: 2026-01-22 06:01:14.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:21.000000000 -0500 Modify: 2026-01-22 06:01:21.000000000 -0500 Change: 2026-01-22 06:01:21.000000000 -0500 Birth: 2026-01-22 06:01:20.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:27.000000000 -0500 Modify: 2026-01-22 06:01:27.000000000 -0500 Change: 2026-01-22 06:01:27.000000000 -0500 Birth: 2026-01-22 06:01:26.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:27.000000000 -0500 Modify: 2026-01-22 06:01:27.000000000 -0500 Change: 2026-01-22 06:01:27.000000000 -0500 Birth: 2026-01-22 06:01:26.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:33.000000000 -0500 Modify: 2026-01-22 06:01:33.000000000 -0500 Change: 2026-01-22 06:01:33.000000000 -0500 Birth: 2026-01-22 06:01:32.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:01:33.000000000 -0500 Modify: 2026-01-22 06:01:33.000000000 -0500 Change: 2026-01-22 06:01:33.000000000 -0500 Birth: 2026-01-22 06:01:32.000000000 -0500 PASS 54 (28s) == sanityn test 55a: rename vs unlink target dir ========= 06:01:42 (1769079702) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (10s) == sanityn test 55b: rename vs unlink source dir ========= 06:01:52 (1769079712) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 06:02:03 (1769079723) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 06:02:18 (1769079738) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 06:02:31 (1769079751) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 06:02:33 (1769079753) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (17s) == sanityn test 55g: rename: race with trylock =========== 06:02:50 (1769079770) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (17s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 06:03:07 (1769079787) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1769079788 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.813854 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (80s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 06:04:27 (1769079867) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1769079872 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.87806 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (74s) == sanityn test 60: Verify data_version behaviour ======== 06:05:42 (1769079942) PASS 60 (5s) == sanityn test 70a: cd directory && rm directory ======== 06:05:47 (1769079947) PASS 70a (5s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:05:52 (1769079952) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:05:59 (1769079959) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00402867 s, 16.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:06:01 (1769079961) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00997112 s, 6.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:06:03 (1769079963) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:06:05 (1769079965) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0142183 s, 4.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:06:07 (1769079967) llite.lustre-ffff8bd6c9830000.xattr_cache=1 llite.lustre-ffff8bd6d5282800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:06:11 (1769079971) llite.lustre-ffff8bd6c9830000.xattr_cache=1 llite.lustre-ffff8bd6d5282800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:06:16 (1769079976) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0146414 s, 69.9 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00834119 s, 123 kB/s 228791: get lock2 228792: get lock1 228792: done 228792: get lock2 228791: done 228791: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 228792: cannot get lock2: Resource deadlock avoided 228792: exit rc=0 228791: done 228791: exit rc=0 PASS 74 (8s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:06:24 (1769079984) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0340904 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232489 s, 45.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229373 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 06:06:33 (1769079993) opening files: ........ closing files: ........ PASS 76 (42s) == sanityn test 77a: check FIFO NRS policy =============== 06:07:15 (1769080035) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257378 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466232 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648558 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134499 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173899 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16471 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147251 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156277 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166228 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193018 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224132 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216166 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216415 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197527 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216987 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198023 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235317 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368933 s, 28.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348124 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445042 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453488 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035032 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534105 s, 19.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628061 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426666 s, 24.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576708 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461939 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615588 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455734 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589355 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536439 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339629 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062141 s, 16.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (6s) == sanityn test 77b: check CRR-N NRS policy ============== 06:07:21 (1769080041) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300172 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378537 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563529 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720983 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104644 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10196 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113735 s, 9.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13646 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130782 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146499 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161132 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164244 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16943 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153636 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146348 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145668 s, 7.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162229 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231687 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235121 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386755 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364597 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334577 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232334 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493461 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245338 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458926 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415529 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00931944 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426468 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490515 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397389 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510632 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228125 s, 460 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.221066 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580719 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685102 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545918 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070608 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741834 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932309 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927796 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805039 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120728 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140915 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131657 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157675 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144892 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142034 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150569 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103817 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400593 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194877 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413784 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255677 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024921 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336692 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278894 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343527 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302824 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03086 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533124 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152085 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567176 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347998 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037948 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414379 s, 25.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (6s) == sanityn test 77c: check ORR NRS policy ================ 06:07:27 (1769080047) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191865 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478581 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851219 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104619 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844132 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10552 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999587 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12084 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939135 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906263 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904457 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535921 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670399 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132565 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126175 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100567 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112741 s, 9.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513773 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271725 s, 386 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140051 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116754 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799619 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727942 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060032 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671493 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673705 s, 15.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517861 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325172 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00984284 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100035 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312961 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302933 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451014 s, 23.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.285432 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029136 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410457 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778506 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108412 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111312 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109835 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104233 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154032 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12674 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147293 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127187 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146662 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165168 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137994 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14657 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165196 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318146 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028681 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019924 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177235 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182193 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00328604 s, 319 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103128 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313504 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358979 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273159 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243442 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329185 s, 31.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491371 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324778 s, 323 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00576458 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00680738 s, 154 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.238275 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050324 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473237 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562514 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541201 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097248 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11818 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114599 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104009 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145469 s, 7.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134007 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851631 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174432 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188626 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179769 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191418 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197444 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242344 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261452 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278247 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230346 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332068 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342297 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302527 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210765 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338336 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464773 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333601 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316197 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335456 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321223 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368762 s, 28.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (8s) == sanityn test 77d: check TRR nrs policy ================ 06:07:35 (1769080055) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230377 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679608 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634081 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822686 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112113 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131688 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11882 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111116 s, 9.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127067 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112607 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114982 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948948 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151921 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141763 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149846 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136677 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102939 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259658 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177274 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272036 s, 38.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305468 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007868 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265116 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284515 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328335 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327822 s, 32.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902288 s, 116 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598266 s, 175 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370965 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350578 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329327 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038383 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731561 s, 143 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.189292 s, 88.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333989 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297763 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234487 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407428 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508762 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642447 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438333 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600696 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117388 s, 89.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148362 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143355 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270763 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359615 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166744 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166885 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175275 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534243 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335349 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398688 s, 26.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332276 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203216 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602288 s, 174 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163801 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111832 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206177 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067849 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00691132 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236383 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487073 s, 215 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02101 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106696 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679949 s, 154 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.170816 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479801 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650945 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703718 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086532 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984903 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780639 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801918 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110284 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675852 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763734 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895007 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117334 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915628 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897259 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937505 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121625 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142591 s, 73.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146272 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273152 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232188 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197547 s, 53.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313187 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316513 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210962 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417763 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225908 s, 46.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339843 s, 30.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341272 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331207 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268002 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024372 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280824 s, 37.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 06:07:43 (1769080063) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.45@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276255 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345172 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051284 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552205 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655518 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100026 s, 10.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145841 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149071 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109861 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138691 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136881 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17991 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170558 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184486 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198006 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200334 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210349 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143922 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196643 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15475 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148454 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168621 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195234 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155277 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17461 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178578 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204361 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191278 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201547 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20262 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210141 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168676 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213664 s, 4.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.232179 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291605 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354748 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706062 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129061 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139395 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137007 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151614 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158593 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139126 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168122 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164513 s, 6.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152303 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15566 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148385 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140084 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157012 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172759 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181485 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182222 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179279 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185558 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196803 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189087 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185926 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184027 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195054 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190044 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196724 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195411 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19551 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195749 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195704 s, 5.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.218058 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285894 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571528 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680337 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152325 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185188 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175434 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153161 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200732 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174116 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162983 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190817 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200084 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226739 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224968 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242042 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237577 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279788 s, 375 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00234443 s, 447 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834075 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244619 s, 429 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055456 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528366 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784118 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414933 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668443 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041905 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385596 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391906 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337316 s, 31.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305707 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276177 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420597 s, 24.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227543 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423003 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567222 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720177 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845511 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698195 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931202 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663133 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958496 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962412 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877403 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690597 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946257 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118832 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107693 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125811 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114316 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235107 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187922 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303819 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234638 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197406 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020361 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157701 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271681 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230502 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222702 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220167 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451685 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038712 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243733 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302888 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319141 s, 32.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (12s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:07:55 (1769080075) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163749 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368593 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381573 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113127 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102553 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112381 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126785 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121287 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122646 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127203 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108902 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141217 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136084 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134279 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121496 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129709 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215153 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6d5282800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112921 s, 92.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502473 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237649 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179303 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266015 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159848 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479604 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121764 s, 86.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104768 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00509304 s, 206 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287769 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310302 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977051 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00230022 s, 456 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888032 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00328419 s, 319 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193252 s, 86.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472337 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585727 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549699 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679856 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512368 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067817 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723093 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674406 s, 15.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590019 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12092 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106549 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137274 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145045 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136804 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161887 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158987 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263836 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229014 s, 45.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450486 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266491 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269627 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274516 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338281 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326741 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364684 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376873 s, 27.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308893 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031859 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423415 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040915 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394188 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463903 s, 22.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201346 s, 83.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526987 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627349 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557763 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622508 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751891 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697161 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746568 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856668 s, 12.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118558 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114763 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147185 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136183 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105388 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122096 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13934 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17376 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488674 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372861 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283372 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121787 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184445 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437706 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167277 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203244 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449698 s, 233 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254045 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230103 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799841 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051428 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698617 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313521 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579199 s, 18.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=2 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191796 s, 87.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530016 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567406 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712666 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584114 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543056 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761765 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913717 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07857 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759535 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137892 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157854 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184115 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161374 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155923 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167329 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6d5282800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216395 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291058 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287165 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285289 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032411 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459863 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041024 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334379 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349893 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365451 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043247 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414668 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420131 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420765 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531349 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539016 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490424 s, 21.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (11s) == sanityn test 77g: Change TBF type directly ============ 06:08:06 (1769080086) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:08:11 (1769080091) oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg345-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg345-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg345-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg345-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 PASS 77h (6s) == sanityn test 77i: Change rank of TBF rule ============= 06:08:17 (1769080097) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (12s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:08:29 (1769080109) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.268744 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386884 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345154 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838916 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937845 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633688 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959105 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808439 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944407 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824482 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912913 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077677 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953688 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856164 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862743 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201929 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218249 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229767 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241151 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311171 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254262 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025681 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037844 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030435 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031829 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332374 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489252 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486179 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371369 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375209 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041317 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037402 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478381 s, 21.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13621 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4905 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (47s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:09:16 (1769080156) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23699 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375232 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618975 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530665 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614595 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271757 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588016 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505425 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787874 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151688 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151638 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176851 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171434 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207212 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207154 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210156 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238544 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281586 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264788 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349641 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387941 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317978 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041175 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423712 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435153 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319584 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476717 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384143 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356466 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488662 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357282 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469225 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405382 s, 25.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6938 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0149 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2614 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792449 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315078 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818952 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063343 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115932 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118104 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118367 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120288 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822795 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129708 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143688 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126998 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126852 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796488 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158437 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163139 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466563 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311934 s, 33.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521468 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548445 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601719 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581731 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054741 s, 19.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631653 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538792 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586091 s, 17.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624416 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577257 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634401 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678748 s, 15.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647314 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829739 s, 12.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.79 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9455 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (140s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 06:11:36 (1769080296) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25459 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662356 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443561 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110795 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119122 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105015 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949337 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843064 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153711 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156617 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133684 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129704 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192311 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149341 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206474 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199857 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231178 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269171 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334004 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398408 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270309 s, 38.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418026 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426849 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283188 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310331 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368867 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413172 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366951 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327134 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496997 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379211 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485606 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475586 s, 22.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.827 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.937 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23741 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455014 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601528 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615773 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692782 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741385 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756094 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542796 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636322 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153736 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153153 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164587 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177682 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177777 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204183 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212495 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210499 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309243 s, 33.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341368 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289154 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364921 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288408 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298707 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038311 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423511 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384648 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314664 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401942 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516247 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441006 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050071 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453584 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461613 s, 22.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8426 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9689 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (132s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:13:48 (1769080428) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.285606 s, 58.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323592 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299985 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890106 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093937 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103796 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966719 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836273 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10958 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106864 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986255 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923899 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102733 s, 10.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102707 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10942 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101803 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132496 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203319 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463934 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267392 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243855 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297063 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313914 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265243 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375919 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288827 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236407 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426785 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557574 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574153 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697924 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339634 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382438 s, 27.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25695 s, 19.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80285 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.268886 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484649 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501269 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578899 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426432 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548701 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529607 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452219 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973803 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103424 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878084 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088226 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110296 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226127 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227239 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278994 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292462 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306929 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274073 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027961 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291916 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331326 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228441 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439859 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401413 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369287 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370359 s, 28.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339789 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350138 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391074 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345949 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378478 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436762 s, 24.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02727 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79997 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25692 s, 65.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327805 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526143 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443424 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298555 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614715 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649172 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767126 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821059 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678628 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076795 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884292 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112874 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114547 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108175 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100059 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968494 s, 10.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026599 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156747 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284793 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327963 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172479 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315107 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216728 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350698 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217232 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208183 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305359 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269975 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268081 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290008 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291969 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303115 s, 34.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99603 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92299 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46622 s, 36.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297214 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490445 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938368 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739573 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809061 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903976 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121792 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102824 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120843 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103732 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117435 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124031 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261273 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273569 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315614 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342721 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106425 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302655 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166045 s, 63.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173606 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381305 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568932 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529404 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634675 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740142 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236714 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257671 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940627 s, 111 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218078 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187855 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181358 s, 57.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150738 s, 69.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88566 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81334 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00378 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76412 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.863449 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018244 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727752 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445604 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609845 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748506 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535937 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686255 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888164 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144134 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171718 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171981 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182115 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20145 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176571 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204624 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216205 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110862 s, 94.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162389 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426835 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459084 s, 228 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043286 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409399 s, 256 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409254 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172091 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842976 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202277 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287654 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036268 s, 289 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233425 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414465 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458674 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156086 s, 67.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5519 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.513 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.287912 s, 58.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294777 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403972 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111487 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114615 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126335 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10887 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119269 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128397 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124736 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147617 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124431 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132997 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14049 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124746 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166991 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145471 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137763 s, 76.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108085 s, 97.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277296 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018266 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196901 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181013 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190555 s, 55.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228452 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210689 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216484 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223034 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302371 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233782 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259638 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247482 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280018 s, 37.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08333 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82172 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463925 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326336 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550997 s, 19.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466478 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146764 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13863 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13255 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134786 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155749 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151034 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132399 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119049 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131174 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177822 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19771 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23655 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222869 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141316 s, 74.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215986 s, 48.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302342 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176657 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220019 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255152 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280235 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289812 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304016 s, 34.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293904 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308801 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316175 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326063 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332147 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334408 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381731 s, 27.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83643 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77634 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86342 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79073 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (366s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 06:19:54 (1769080794) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310059 s, 54.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521919 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529243 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677772 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10361 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104881 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116117 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103774 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121836 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128529 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110508 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153597 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156237 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124448 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170788 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170594 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16208 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193107 s, 54.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448799 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258295 s, 40.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355581 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287522 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402092 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404894 s, 25.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324447 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351096 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360962 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482517 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600388 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417216 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463925 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450603 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424509 s, 24.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04609 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79816 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (40s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 06:20:34 (1769080834) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.321337 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331558 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178473 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040546 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523624 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757267 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820769 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864273 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687978 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561964 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987211 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959799 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100994 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214331 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223998 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238714 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23783 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027913 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408569 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343205 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309307 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164798 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182818 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235887 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303766 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247272 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259598 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268673 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262334 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445367 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265047 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457079 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498159 s, 210 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05371 s, 20.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80898 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (42s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 06:21:16 (1769080876) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.330889 s, 50.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516508 s, 20.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389477 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703501 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057254 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058533 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646082 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512759 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631768 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101716 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14287 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105899 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129012 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147065 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16119 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174703 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189293 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117141 s, 89.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137294 s, 76.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141622 s, 74.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361191 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109689 s, 95.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183545 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025932 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021416 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066054 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381837 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273148 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281356 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294689 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419339 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379614 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428759 s, 245 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15661 s, 20.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97029 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (35s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 06:21:51 (1769080911) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467601 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263209 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400915 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579203 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560215 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784204 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103511 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943807 s, 11.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092861 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113209 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123587 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140289 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158944 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155856 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155608 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214848 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216194 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212118 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014301 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168291 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236677 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174317 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233291 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253297 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021582 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233084 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313603 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331999 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265224 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271633 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396319 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329928 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285536 s, 36.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88975 s, 10.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78341 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94775 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73838 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (78s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 06:23:09 (1769080989) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.869062 s, 19.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332634 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370791 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203569 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534938 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558443 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456411 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538447 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698197 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154914 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180388 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203254 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189733 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209009 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223979 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22192 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238595 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01594 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020231 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259747 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019204 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253345 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281968 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222936 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223343 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254033 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256935 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207737 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214809 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253995 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257068 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299167 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268002 s, 39.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7802 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9405 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 06:24:15 (1769081055) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.63258 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042068 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616055 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640577 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162873 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147958 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179362 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203876 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214122 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208244 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229828 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202295 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220184 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229529 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314517 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292212 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370502 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143631 s, 73.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329064 s, 319 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915672 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130807 s, 80.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129308 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131513 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599214 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631569 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898271 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913682 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391031 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532951 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318318 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00998281 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102974 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111765 s, 93.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5785 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89418 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg345-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.973248 s, 17.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091004 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097304 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113748 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21826 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254892 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194382 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265439 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195371 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264551 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289644 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200001 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29194 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2136 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296527 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329997 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345427 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132728 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102681 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214796 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141913 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530436 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128157 s, 81.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104888 s, 100 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08229 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130776 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260886 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00235895 s, 445 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026053 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23388 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031968 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130614 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134099 s, 7.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.608 s, 7.2 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92172 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1039 s, 10.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89174 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (141s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 06:26:37 (1769081197) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre opts: Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg345-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg345-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.45,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg345-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.45,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg345-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.45,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/59629/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.45,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg345-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Started clients oleg345-client.virtnet: 192.168.203.145@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Started clients oleg345-client.virtnet: 192.168.203.145@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bd6c915d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8bd6fb1c7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6c915d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6fb1c7800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.420618 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773241 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560691 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982925 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131114 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157576 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136998 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172104 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15692 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194569 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1427 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138458 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1972 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164082 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272428 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209135 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244403 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223021 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562146 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570047 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577877 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291733 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708877 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307047 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289551 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556654 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356578 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484086 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408504 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042726 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610125 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502677 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831873 s, 12.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.64616 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16687 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482799 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741009 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759893 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105686 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147659 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128203 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191917 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176483 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240711 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194392 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173425 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234554 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280007 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255962 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289825 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254934 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294608 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6c915d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108854 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077532 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108412 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113435 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084796 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907002 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900049 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131368 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117829 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119673 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118975 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125844 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129793 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121208 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126576 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133024 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6c915d800.lock_unused_count=4 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.3015 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90626 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (171s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 06:29:29 (1769081369) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg345-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (13s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:29:42 (1769081382) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (7s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:29:49 (1769081389) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg345-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.69159 s, 284 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg345-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1604 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (60s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:30:50 (1769081450) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.342811 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707762 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113393 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118794 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986757 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434993 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154453 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071059 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178956 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607652 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168498 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298187 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261921 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30109 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341923 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378365 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10113 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896215 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766198 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093188 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405592 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394853 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6c915d800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6c915d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354426 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113719 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133647 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208865 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123522 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161631 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489508 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589949 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412774 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348899 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619572 s, 16.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg345-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg345-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48167 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13821 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.431628 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140473 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135158 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134692 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056841 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22143 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193049 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321313 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331142 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323187 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35321 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35937 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383318 s, 2.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354084 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2623 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265522 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288007 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8bd6c915d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8bd6c915d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183981 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186843 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20566 s, 5.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247271 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192347 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174098 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226207 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185021 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202205 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236133 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221693 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194344 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205911 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22754 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216472 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216329 s, 4.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg345-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg345-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.28062 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14756 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (79s) == sanityn test 77o: Changing rank should not panic ====== 06:32:09 (1769081529) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (9s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:32:18 (1769081538) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (72s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:33:30 (1769081610) mds.MDS.mdt.nrs_policies=tbf oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg345-client: oleg345-server: ssh exited with exit code 36 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 oleg345-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg345-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg345-client: oleg345-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (17s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 06:33:47 (1769081627) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 268299 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:34:27 (1769081667) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 06:34:31 (1769081671) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826934965 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:34:32.000000000 -0500 Modify: 2026-01-22 06:34:32.000000000 -0500 Change: 2026-01-22 06:34:32.000000000 -0500 Birth: 2026-01-22 06:34:31.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:34:44 (1769081684) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:34:46 (1769081686) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:34:47 (1769081687) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:34:49 (1769081689) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:34:50 (1769081690) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 06:34:51 (1769081691) - open/close 2291 (time 1769081703.52 total 10.00 last 229.08) total: 2500 open/close in 10.98 seconds: 227.62 ops/second snapshot_time 1769081726.734939047 secs.nsecs start_time 1769081691.777519264 secs.nsecs elapsed_time 34.957419783 secs.nsecs open 2505 samples [usecs] 106 4434 1487538 1009011954 close 2504 samples [usecs] 68 2486 508605 122282173 mknod 2501 samples [usecs] 193 4321 1307718 791641204 unlink 2500 samples [usecs] 277 45305 2203308 9479623078 mkdir 2 samples [usecs] 857 1016 1873 1766705 rmdir 2 samples [usecs] 2198 4999 7197 29821205 rename 2500 samples [usecs] 288 10944 4453657 12841405991 getattr 3763 samples [usecs] 6 1414 90463 5792375 parallel_rename_file 2500 samples [usecs] 288 10944 4453657 12841405991 crossdir_rename 2500 samples [usecs] 288 10944 4453657 12841405991 PASS 81d (39s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 06:35:30 (1769081730) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7531 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7531 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 06:35:33 (1769081733) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 06:35:35 (1769081735) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 06:35:43 (1769081743) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1769081743 Finishing test test1 at 1769081744 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 06:35:46 (1769081746) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 06:35:47 (1769081747) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 06:35:49 (1769081749) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 06:35:50 (1769081750) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5745 0x1671 0x240000400 1 5452 0x154c 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5746 0x1672 0x240000400 1 5453 0x154d 0x280000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 06:36:00 (1769081760) osc.lustre-OST0000-osc-ffff8bd6c915d800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8bd6fb1c7800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8bd6c915d800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8bd6fb1c7800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0632128 s, 130 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09578 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 285218 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0284686 s, 144 kB/s osc.lustre-OST0000-osc-ffff8bd6c915d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8bd6fb1c7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6c915d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd6fb1c7800.idle_timeout=debug PASS 94 (12s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 06:36:12 (1769081772) llite.lustre-ffff8bd6c915d800.fast_read=0 llite.lustre-ffff8bd6fb1c7800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0249785 s, 84.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8bd6c915d800.fast_read=1 llite.lustre-ffff8bd6fb1c7800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 06:36:27 (1769081787) llite.lustre-ffff8bd6c915d800.fast_read=0 llite.lustre-ffff8bd6fb1c7800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0043564 s, 2.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8bd6c915d800.fast_read=1 llite.lustre-ffff8bd6fb1c7800.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 06:36:36 (1769081796) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 06:36:38 (1769081798) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0023791 s, 1.7 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 06:36:41 (1769081801) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.063834 s, 32.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 06:36:44 (1769081804) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0678107 s, 30.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 06:36:46 (1769081806) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00196422 s, 6.1 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 06:36:49 (1769081809) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00712648 s, 575 kB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 06:36:51 (1769081811) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00234516 s, 1.7 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 06:36:54 (1769081814) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00366678 s, 1.1 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 06:36:58 (1769081818) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1769081819 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1769081822 Finishing test test23 at 1769081822 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 06:37:06 (1769081826) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 06:37:07 (1769081827) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 06:37:25 (1769081845) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 06:37:26 (1769081846) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00260448 s, 393 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937500 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:37:27.000000000 -0500 Modify: 2026-01-22 06:37:27.000000000 -0500 Change: 2026-01-22 06:37:27.000000000 -0500 Birth: 2026-01-22 06:37:26.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937500 -rw-r--r-- 1 0 0 1769081846 1769081847 1769081847 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937500 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:37:27.000000000 -0500 Modify: 2026-01-22 06:37:27.000000000 -0500 Change: 2026-01-22 06:37:27.000000000 -0500 Birth: 2026-01-22 06:37:26.000000000 -0500 1769081847 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 06:37:29 (1769081849) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 06:37:31 (1769081851) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.109835 s, 95.5 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5459 0x1553 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 06:37:36 (1769081856) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0995822 s, 105 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5755 0x167b 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7531 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 06:37:44 (1769081864) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00312114 s, 3.2 MB/s llite.lustre-ffff8bd6c915d800.hybrid_io=0 llite.lustre-ffff8bd6fb1c7800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.29611 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 06:37:51 (1769081871) 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 192.168.203.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 PASS 109 (520s) == sanityn test 110: do not grant another lock on resend ========================================================== 06:46:31 (1769082391) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 06:46:32.000000000 -0500 Modify: 2026-01-22 06:46:32.000000000 -0500 Change: 2026-01-22 06:46:32.000000000 -0500 Birth: 2026-01-22 06:46:31.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg345-client.virtnet oleg345-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 06:47:36 (1769082456) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 06:47:38 (1769082458) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 06:47:39 (1769082459) lustre-OST0000-osc-ffff8bd6c915d800 active. lustre-OST0001-osc-ffff8bd6c915d800 active. lustre-OST0000-osc-ffff8bd6c915d800 active. lustre-OST0001-osc-ffff8bd6c915d800 active. MGC192.168.203.145@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 06:47:41 (1769082461) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 06:47:42 (1769082462) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 06:47:43 (1769082463) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 06:47:44 (1769082464) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00478791 s, 4.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0156955 s, 1.3 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 06:47:49 (1769082469) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0912 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.09 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0988 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0943 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0957 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0948 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0976 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1368 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1493 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1361 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1429 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.14 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1519 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1559 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2262 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2405 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2234 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2429 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bd6c915d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd6c915d800.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 9330 sec ============== 06:48:56 (1769082536) === sanityn: start cleanup 06:48:57 (1769082537) === Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg345-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 06:49:49 (1769082589) ===