-----============= acceptance-small: sanityn ============----- Tue Apr 1 03:44:52 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg611-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 28 102 skipping tests SLOW=no: 33a === sanityn: start setup 03:45:42 (1743493542) === Starting client oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Started clients oleg611-client.virtnet: 192.168.206.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project) oleg611-client.virtnet: executing check_config_client /mnt/lustre oleg611-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg611-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f0086354000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f0086554000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0086354000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0086554000.idle_timeout=debug disable quota as required oleg611-server: oleg611-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 03:47:02 (1743493622) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896652 s, 11.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7365] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 03:47:08 (1743493628) PASS 0 (23s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 03:47:31 (1743493651) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (20s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 03:47:52 (1743493672) -rw-r--r-- 1 root root 0 Apr 1 03:47 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (19s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 03:48:12 (1743493692) -rw-r--r-- 1 root root 0 Apr 1 03:48 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (19s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 03:48:32 (1743493712) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (18s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 03:48:50 (1743493730) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (18s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 03:49:08 (1743493748) total 1 -rwxrwxrwx 1 root root 0 Apr 1 03:47 f2a -rwxrwxrwx 1 root root 0 Apr 1 03:48 f2b total 1 -rwxrwxrwx 1 root root 0 Apr 1 03:47 f2a -rwxrwxrwx 1 root root 0 Apr 1 03:48 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0290192 s, 17.6 kB/s PASS 2e (20s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 03:49:29 (1743493769) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (8s) == sanityn test 2g: check blocks update on sync write ==== 03:49:38 (1743493778) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 1.18039 s, 1.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (21s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 03:50:00 (1743493800) PASS 3 (18s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 03:50:18 (1743493818) PASS 4 (19s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 03:50:36 (1743493836) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (18s) == sanityn test 6: remove of open file on other node ============================================================================ 03:50:54 (1743493854) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (17s) == sanityn test 7: remove of open directory on other node ======================================================================= 03:51:12 (1743493872) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (20s) == sanityn test 8: remove of open special file on other node ==================================================================== 03:51:31 (1743493891) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (18s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 03:51:49 (1743493909) PASS 9a (18s) == sanityn test 9b: append to striped sparse file ======== 03:52:07 (1743493927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151047 s, 6.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.112086 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (18s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 03:52:26 (1743493946) 1+0 records in 1+0 records out 1 byte copied, 0.00818168 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0349238 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0329894 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204007 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.01591 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0344434 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0456362 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0446275 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0234097 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0647417 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0181863 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.041981 s, 0.0 kB/s PASS 10a (21s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 03:52:48 (1743493968) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0317695 s, 96.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.124906 s, 32.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000425599 s, 7.2 MB/s PASS 10b (19s) == sanityn test 11: execution of file opened for write should return error ============================================================== 03:53:07 (1743493987) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (17s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 03:53:24 (1743494004) warning: '-runtime' deprecated, use '-t runtime' instead - link 219 (time 1743494020.11 total 10.04 last 21.81) - link 436 (time 1743494030.12 total 20.05 last 21.69) - link 646 (time 1743494040.16 total 30.09 last 20.91) - link 856 (time 1743494050.25 total 40.17 last 20.82) - link 1053 (time 1743494060.27 total 50.20 last 19.65) - link 1237 (time 1743494070.31 total 60.24 last 18.33) - link 1430 (time 1743494080.31 total 70.24 last 19.29) - link 1632 (time 1743494090.32 total 80.25 last 20.19) - link 1834 (time 1743494100.35 total 90.28 last 20.13) total: 2019 link in 100.02 seconds: 20.19 ops/second using seed 1755501885 running for 100 seconds - stat 2668 (time 1743494022 ; total 11 ; last 11) - stat 5506 (time 1743494033 ; total 22 ; last 11) - stat 8748 (time 1743494044 ; total 33 ; last 11) - stat 10000 (time 1743494049 ; total 38 ; last 5) - stat 13604 (time 1743494060 ; total 49 ; last 11) - stat 15174 (time 1743494071 ; total 60 ; last 11) - stat 16365 (time 1743494082 ; total 71 ; last 11) - stat 17750 (time 1743494093 ; total 82 ; last 11) - stat 19140 (time 1743494104 ; total 93 ; last 11) - stat 20000 (time 1743494110 ; total 99 ; last 6) total: 20134 stats in 100 seconds: 201.339996 stats/second - unlinked 0 (time 1743494061 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2019) error: No such file or directory total: 2018 unlinks in 94 seconds: 21.468084 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21213) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21215) - No such process PASS 12 (167s) == sanityn test 13: test directory page revocation ======= 03:56:11 (1743494171) f13 f13 PASS 13 (23s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 03:56:34 (1743494194) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22407 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (18s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 03:56:52 (1743494212) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22988 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (17s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 03:57:10 (1743494230) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 23567 Terminated $DIR1/$tdir/sleep 60 PASS 14b (17s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 03:57:27 (1743494247) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 24151 Terminated $DIR1/$tdir/sleep 60 PASS 14c (16s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 03:57:44 (1743494264) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 24737 Terminated $DIR1/$tdir/sleep 60 PASS 14d (22s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 03:58:06 (1743494286) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (66s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 03:59:13 (1743494353) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.61559 s, 5.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.16358 s, 7.2 MB/s Chance of close/open is 1 in 50 Seed set to 4363 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[0] 1743494365.276791 mapwrite 0x35e559 thru 0x364158 (0x5c00 bytes) 000009[0] 1743494365.947594 mapread 0x14aeb3 thru 0x154980 (0x9ace bytes) 000011[0] 1743494366.000539 read 0x197251 thru 0x19a897 (0x3647 bytes) 000013[1] 1743494366.076437 write 0x0948cd thru 0x094bc6 (0x2fa bytes) truncating to largest ever: 0x3ead1a 000014[1] 1743494366.161448 trunc from 0x364159 to 0x3ead19 (0x86bc1 bytes) 000019[0] 1743494366.346758 mapwrite 0x3e7258 thru 0x3f00f5 (0x8e9e bytes) 000020[0] 1743494366.613582 write 0x6999cc thru 0x69cb75 (0x31aa bytes) 000022[0] 1743494366.682590 trunc from 0x69cb76 to 0x226d54 (0x475e21 bytes) 000027[1] 1743494366.787144 write 0x4b5b59 thru 0x4ba4c0 (0x4968 bytes) 000029[0] 1743494366.846530 mapwrite 0x77e4d6 thru 0x78962b (0xb156 bytes) 000030[1] 1743494367.266955 mapwrite 0x6c3ee6 thru 0x6cce45 (0x8f60 bytes) 000037[0] 1743494367.850681 trunc from 0x78962c to 0x0c63b8 (0x6c3273 bytes) 000038[1] 1743494367.967080 mapread 0x04258d thru 0x0455bc (0x3030 bytes) 000039[0] 1743494368.051989 mapread 0x07b31d thru 0x089214 (0xdef8 bytes) truncating to largest ever: 0x8925ee 000041[1] 1743494368.164897 trunc from 0x0c63b9 to 0x8925ed (0x7cc235 bytes) 000042[0] 1743494368.331126 mapread 0x4c6dbb thru 0x4cd186 (0x63cc bytes) 000044[0] 1743494368.512889 read 0x56818c thru 0x570d8b (0x8c00 bytes) 000046[0] 1743494368.617056 read 0x78405e thru 0x7919a6 (0xd949 bytes) 000047[1] 1743494368.639709 read 0x68ffc5 thru 0x69b5ef (0xb62b bytes) 000048[0] 1743494368.709275 mapread 0x09b16c thru 0x09c106 (0xf9b bytes) 000050[0] 1743494368.801785 read 0x244dac thru 0x24fc0d (0xae62 bytes) 000051[0] 1743494368.831213 write 0x498b51 thru 0x49cf9f (0x444f bytes) 000054[0] 1743494369.038544 trunc from 0x8925ee to 0x575e6a (0x31c783 bytes) 000057[0] 1743494369.112898 mapread 0x3c27fb thru 0x3c2ceb (0x4f1 bytes) 000059[1] 1743494369.240152 trunc from 0x575e6b to 0x2231ce (0x352c9c bytes) 000060[0] 1743494369.425765 mapread 0x222fb2 thru 0x2231ce (0x21d bytes) 000061[0] 1743494369.576823 read 0x1fec49 thru 0x207085 (0x843d bytes) 000062[1] 1743494369.962030 read 0x019ac2 thru 0x019f08 (0x447 bytes) 000064[0] 1743494370.443728 mapwrite 0x76fac3 thru 0x772f23 (0x3461 bytes) 000065[1] 1743494371.677302 read 0x25dd2f thru 0x26ba6a (0xdd3c bytes) 000067[1] 1743494372.375628 mapwrite 0x2f5b78 thru 0x2fdd77 (0x8200 bytes) 000069[0] 1743494373.021990 read 0x657118 thru 0x664010 (0xcef9 bytes) 000070[0] 1743494373.064950 read 0x6074be thru 0x610c2c (0x976f bytes) 000072[1] 1743494373.131692 mapread 0x48a77a thru 0x492b6a (0x83f1 bytes) 000074[1] 1743494373.614170 mapwrite 0x0e94aa thru 0x0ec814 (0x336b bytes) 000075[1] 1743494373.784073 write 0x8d11d3 thru 0x8d3376 (0x21a4 bytes) 000077[1] 1743494373.839817 write 0x0fde1e thru 0x10a8ba (0xca9d bytes) 000079[1] 1743494373.939930 mapread 0x689cbd thru 0x69427d (0xa5c1 bytes) 000081[1] 1743494373.980231 mapwrite 0x53bb29 thru 0x53d8fc (0x1dd4 bytes) 000083[0] 1743494374.213959 mapread 0x5fe807 thru 0x606bbf (0x83b9 bytes) 000084[0] 1743494374.376164 trunc from 0x8d3377 to 0x524ea7 (0x3ae4cf bytes) 000085[1] 1743494374.529639 mapread 0x356851 thru 0x36459b (0xdd4b bytes) 000089[1] 1743494374.775312 read 0x4c7c3d thru 0x4d3000 (0xb3c4 bytes) 000090[1] 1743494374.826948 mapread 0x4ce07f thru 0x4d7ad1 (0x9a53 bytes) 000091[1] 1743494374.875977 trunc from 0x524ea8 to 0x0e85cf (0x43c8d8 bytes) 000093[1] 1743494374.985995 mapread 0x040cec thru 0x04105c (0x371 bytes) 000094[1] 1743494375.000441 mapread 0x0c475a thru 0x0d3138 (0xe9df bytes) 000095[1] 1743494375.036110 trunc from 0x0e85d0 to 0x521ee4 (0x439915 bytes) 000097[1] 1743494375.111015 write 0x108d5e thru 0x10eb73 (0x5e16 bytes) 000098[0] 1743494375.190570 mapread 0x135803 thru 0x13ce4f (0x764d bytes) 000099[1] 1743494375.308072 mapwrite 0x957be6 thru 0x95a6aa (0x2ac5 bytes) 000103[0] 1743494375.621565 mapwrite 0x1d41fd thru 0x1e1e9c (0xdca0 bytes) 000104[0] 1743494375.912648 mapread 0x8d0995 thru 0x8d6156 (0x57c2 bytes) 000105[1] 1743494376.059870 mapwrite 0x77307d thru 0x778a56 (0x59da bytes) 000106[0] 1743494376.257517 mapwrite 0x52d933 thru 0x53b291 (0xd95f bytes) 000107[1] 1743494376.591262 trunc from 0x95a6ab to 0x2c8d42 (0x691968 bytes) 000108[0] 1743494376.732598 mapread 0x0d49c7 thru 0x0de285 (0x98bf bytes) 000109[1] 1743494376.808319 mapwrite 0x38cd61 thru 0x39664e (0x98ee bytes) 000110[1] 1743494377.052843 trunc from 0x39664f to 0x0497e0 (0x34ce6e bytes) 000111[0] 1743494377.130297 trunc from 0x0497e1 to 0x0d2327 (0x88b47 bytes) 000121[1] 1743494377.420805 read 0x05bc8b thru 0x05ecf6 (0x306c bytes) 000122[1] 1743494377.580189 write 0x0f0ab4 thru 0x0ff358 (0xe8a5 bytes) 000124[1] 1743494377.668208 read 0x089edf thru 0x08c02b (0x214d bytes) 000126[0] 1743494377.968981 mapwrite 0x1da636 thru 0x1e8909 (0xe2d4 bytes) 000128[0] 1743494378.409874 mapread 0x15169f thru 0x153997 (0x22f9 bytes) 000131[1] 1743494378.529194 mapwrite 0x737faf thru 0x741fdd (0xa02f bytes) 000135[0] 1743494379.335915 mapread 0x683baf thru 0x6865fe (0x2a50 bytes) 000140[1] 1743494379.696491 read 0x051f91 thru 0x0595e0 (0x7650 bytes) 000142[1] 1743494379.713264 read 0x2a99e3 thru 0x2b53d8 (0xb9f6 bytes) 000143[0] 1743494379.742034 trunc from 0x741fde to 0x55661d (0x1eb9c0 bytes) 000144[1] 1743494379.867559 read 0x4be313 thru 0x4c599e (0x768c bytes) 000145[0] 1743494379.915718 mapwrite 0x554402 thru 0x562e9d (0xea9c bytes) 000147[0] 1743494380.325126 mapwrite 0x56f8cc thru 0x57a0dc (0xa811 bytes) 000148[1] 1743494380.637628 trunc from 0x57a0dd to 0x65106b (0xd6f8f bytes) 000151[1] 1743494380.802080 write 0x9cdd57 thru 0x9d2124 (0x43ce bytes) 000152[0] 1743494380.809738 read 0x77d71e thru 0x78135e (0x3c41 bytes) 000153[0] 1743494380.928722 mapwrite 0x4e49a0 thru 0x4f3e41 (0xf4a2 bytes) 000154[1] 1743494381.265478 mapwrite 0x54f415 thru 0x5529ba (0x35a6 bytes) 000155[0] 1743494381.513279 mapwrite 0x9f42a2 thru 0x9fffff (0xbd5e bytes) 000158[1] 1743494381.881052 mapwrite 0x79a01b thru 0x7a99ef (0xf9d5 bytes) 000161[0] 1743494382.339771 mapread 0x143d16 thru 0x146e35 (0x3120 bytes) 000162[0] 1743494382.417592 read 0x38444c thru 0x38db21 (0x96d6 bytes) 000163[0] 1743494382.454519 write 0x79332c thru 0x79f6a2 (0xc377 bytes) 000164[1] 1743494382.494958 write 0x0e18c5 thru 0x0efc3e (0xe37a bytes) 000165[0] 1743494382.571501 write 0x6a256c thru 0x6a7376 (0x4e0b bytes) 000166[0] 1743494382.613937 write 0x4870f0 thru 0x489c9f (0x2bb0 bytes) 000167[0] 1743494382.658131 write 0x3b985e thru 0x3c2b4a (0x92ed bytes) truncating to largest ever: 0x907040 000169[1] 1743494382.872240 trunc from 0xa00000 to 0x90703f (0xf8fc0 bytes) 000174[1] 1743494383.101269 mapread 0x708d00 thru 0x717aa6 (0xeda7 bytes) 000177[0] 1743494383.243926 trunc from 0x907040 to 0x8818bc (0x85783 bytes) 000178[1] 1743494383.361829 mapwrite 0x9e7777 thru 0x9f3c90 (0xc51a bytes) 000179[1] 1743494383.776060 mapwrite 0x532b93 thru 0x535832 (0x2ca0 bytes) 000182[1] 1743494384.008746 read 0x8cdc4e thru 0x8d8817 (0xabca bytes) 000183[1] 1743494384.033239 write 0x060183 thru 0x06202f (0x1ead bytes) 000187[0] 1743494384.231225 mapwrite 0x4cded5 thru 0x4cf633 (0x175f bytes) 000190[1] 1743494384.584525 trunc from 0x9f3c91 to 0x2dd8d4 (0x7163bc bytes) 000191[0] 1743494384.724672 write 0x2d7604 thru 0x2dfd10 (0x870d bytes) 000192[1] 1743494384.804311 write 0x5c8f6b thru 0x5ca36d (0x1403 bytes) 000193[1] 1743494384.839602 write 0x49ba83 thru 0x4a7fa4 (0xc522 bytes) 000195[0] 1743494384.933857 mapread 0x0894f1 thru 0x090c4d (0x775d bytes) 000196[0] 1743494385.037976 mapwrite 0x37ffdc thru 0x38e38a (0xe3af bytes) 000197[1] 1743494385.985416 write 0x43cfa6 thru 0x44219e (0x51f9 bytes) 000198[0] 1743494386.075002 trunc from 0x5ca36e to 0x67526b (0xaaefe bytes) 000199[1] 1743494386.182600 mapread 0x345995 thru 0x352f91 (0xd5fd bytes) 000200[0] 1743494386.372723 mapwrite 0x3ff5df thru 0x40a18b (0xabad bytes) 000202[1] 1743494386.916800 read 0x194873 thru 0x19ef9a (0xa728 bytes) 000203[0] 1743494386.992947 mapwrite 0x52542e thru 0x52ef20 (0x9af3 bytes) 000204[1] 1743494387.424796 read 0x0d4f75 thru 0x0d6a21 (0x1aad bytes) 000205[0] 1743494387.435664 mapread 0x21c8e5 thru 0x228c0f (0xc32b bytes) 000208[0] 1743494387.564329 read 0x5931b7 thru 0x59bc7e (0x8ac8 bytes) 000209[1] 1743494387.646801 write 0x2c35ef thru 0x2d0781 (0xd193 bytes) 000210[1] 1743494387.749250 write 0x64f033 thru 0x650319 (0x12e7 bytes) 000212[1] 1743494387.792641 read 0x14b410 thru 0x154054 (0x8c45 bytes) 000216[1] 1743494387.958324 read 0x5996b1 thru 0x59e74d (0x509d bytes) 000217[1] 1743494388.016314 trunc from 0x67526c to 0x0dcd38 (0x598533 bytes) 000219[1] 1743494388.135449 trunc from 0x0dcd39 to 0x7d1d66 (0x6f502e bytes) 000222[0] 1743494388.277954 trunc from 0x7d1d67 to 0x3a31f4 (0x42eb72 bytes) 000225[0] 1743494388.517054 read 0x1ddaad thru 0x1e7e99 (0xa3ed bytes) 000226[0] 1743494388.951752 mapwrite 0x88d85a thru 0x89c143 (0xe8ea bytes) 000227[1] 1743494389.583459 read 0x24a4ba thru 0x253c37 (0x977e bytes) 000229[1] 1743494390.079989 read 0x328756 thru 0x334599 (0xbe44 bytes) 000231[0] 1743494390.115045 mapread 0x075c97 thru 0x07ea79 (0x8de3 bytes) 000233[1] 1743494390.347255 trunc from 0x89c144 to 0x01229f (0x889ea4 bytes) 000235[1] 1743494390.670739 mapwrite 0x580c07 thru 0x589dc7 (0x91c1 bytes) 000237[1] 1743494391.035099 trunc from 0x589dc8 to 0x7abfab (0x2221e4 bytes) 000240[1] 1743494391.087905 mapread 0x77c981 thru 0x780efc (0x457c bytes) 000241[1] 1743494391.253305 mapread 0x0ecee7 thru 0x0f702a (0xa144 bytes) 000244[0] 1743494391.487081 read 0x34066f thru 0x34d5dc (0xcf6e bytes) 000246[0] 1743494391.549828 trunc from 0x7abfac to 0x0314af (0x77aafc bytes) 000249[1] 1743494391.789764 mapwrite 0x7343fe thru 0x73f685 (0xb288 bytes) 000252[1] 1743494392.353261 mapwrite 0x6a732d thru 0x6aa78f (0x3463 bytes) 000253[0] 1743494392.529503 mapwrite 0x222e4b thru 0x224f5d (0x2113 bytes) 000254[0] 1743494392.857339 mapwrite 0x7d52e9 thru 0x7df676 (0xa38e bytes) 000256[0] 1743494393.286478 mapwrite 0x11ebb5 thru 0x12e21a (0xf666 bytes) 000258[1] 1743494393.701523 mapwrite 0x195c6d thru 0x19f453 (0x97e7 bytes) 000259[1] 1743494394.002872 mapwrite 0x5c069f thru 0x5c68fa (0x625c bytes) 000260[0] 1743494394.318441 read 0x46e436 thru 0x4720c6 (0x3c91 bytes) 000262[0] 1743494394.374672 trunc from 0x7df677 to 0x0d42ca (0x70b3ac bytes) 000263[0] 1743494394.510646 mapwrite 0x7f2301 thru 0x7f9dc6 (0x7ac6 bytes) 000264[0] 1743494394.775014 mapread 0x7cc50b thru 0x7d0689 (0x417f bytes) 000266[0] 1743494394.804024 mapread 0x6fbf6a thru 0x700f20 (0x4fb7 bytes) 000268[1] 1743494394.877866 mapread 0x5ca642 thru 0x5d03ac (0x5d6b bytes) 000273[0] 1743494395.132521 mapread 0x2c6965 thru 0x2caf37 (0x45d3 bytes) 000275[0] 1743494395.385118 mapread 0x1d9a67 thru 0x1dc89e (0x2e38 bytes) 000276[1] 1743494395.479385 read 0x3b2762 thru 0x3c0efc (0xe79b bytes) 000277[0] 1743494395.540890 write 0x6401bf thru 0x645305 (0x5147 bytes) 000279[1] 1743494395.671250 trunc from 0x7f9dc7 to 0x8ff440 (0x10567a bytes) 000280[1] 1743494395.828412 mapread 0x03fac2 thru 0x04a266 (0xa7a5 bytes) 000283[1] 1743494396.058314 write 0x778de1 thru 0x77f8b5 (0x6ad5 bytes) 000284[0] 1743494396.108676 trunc from 0x8ff441 to 0x012c34 (0x8ec80c bytes) 000285[0] 1743494396.255063 mapread 0x0022d5 thru 0x004bd3 (0x28ff bytes) 000286[1] 1743494396.370852 mapread 0x0042ec thru 0x012981 (0xe696 bytes) 000288[1] 1743494396.489245 read 0x00af2f thru 0x010f86 (0x6058 bytes) 000289[1] 1743494396.509165 write 0x371fe7 thru 0x37f7b1 (0xd7cb bytes) 000293[1] 1743494396.737539 write 0x0a6fc8 thru 0x0b1b3c (0xab75 bytes) 000300[1] 1743494397.286150 read 0x2b60a6 thru 0x2bdc6b (0x7bc6 bytes) 000306[0] 1743494397.478902 write 0x671e64 thru 0x679bd2 (0x7d6f bytes) 000308[0] 1743494397.543499 write 0x95851a thru 0x95d752 (0x5239 bytes) 000310[1] 1743494397.654552 mapread 0x59acf1 thru 0x59db28 (0x2e38 bytes) 000313[0] 1743494397.868210 mapread 0x2ea84d thru 0x2f8a50 (0xe204 bytes) 000318[1] 1743494398.090937 trunc from 0x95d753 to 0x2c3af9 (0x699c59 bytes) 000319[0] 1743494398.240890 trunc from 0x2c3afa to 0x52e41e (0x26a925 bytes) 000320[0] 1743494398.364762 mapwrite 0x5f3b4a thru 0x5fb218 (0x76cf bytes) 000322[0] 1743494398.668994 mapread 0x3d8764 thru 0x3e1590 (0x8e2d bytes) 000323[0] 1743494398.714509 mapwrite 0x114d4b thru 0x119644 (0x48fa bytes) 000326[0] 1743494399.050985 write 0x9d17a6 thru 0x9d8c42 (0x749d bytes) 000327[0] 1743494399.054424 mapwrite 0x298239 thru 0x2a4d6d (0xcb35 bytes) 000329[0] 1743494399.516665 write 0x31fc92 thru 0x3251f9 (0x5568 bytes) 000331[1] 1743494399.591598 mapwrite 0x887d67 thru 0x8946da (0xc974 bytes) 000336[0] 1743494400.640179 read 0x3ec013 thru 0x3ec683 (0x671 bytes) 000338[1] 1743494400.735640 write 0x64a78d thru 0x656612 (0xbe86 bytes) 000341[0] 1743494401.145422 mapread 0x189aca thru 0x193868 (0x9d9f bytes) 000344[0] 1743494401.464717 mapwrite 0x2522c8 thru 0x252ee7 (0xc20 bytes) 000345[0] 1743494401.794283 read 0x80f382 thru 0x81138b (0x200a bytes) 000346[0] 1743494401.845094 read 0x651a6b thru 0x652b85 (0x111b bytes) 000350[0] 1743494401.934186 read 0x93a4ab thru 0x94a211 (0xfd67 bytes) 000356[1] 1743494402.068230 read 0x9b719e thru 0x9bf965 (0x87c8 bytes) 000357[1] 1743494402.130552 mapread 0x1f49ea thru 0x201669 (0xcc80 bytes) 000358[1] 1743494402.201577 mapwrite 0x44ffa0 thru 0x4507dd (0x83e bytes) 000360[1] 1743494402.485520 write 0x4d2843 thru 0x4d5305 (0x2ac3 bytes) 000364[0] 1743494402.559120 trunc from 0x9d8c43 to 0x081a6d (0x9571d5 bytes) 000365[0] 1743494402.734387 mapread 0x05d298 thru 0x05ea14 (0x177d bytes) 000367[0] 1743494402.820608 write 0x9d6023 thru 0x9df9c9 (0x99a7 bytes) 000368[0] 1743494402.836532 mapread 0x7f1a4e thru 0x7f4205 (0x27b8 bytes) 000369[1] 1743494402.901568 mapread 0x2fb269 thru 0x309d8d (0xeb25 bytes) 000370[0] 1743494403.065127 mapread 0x2d76a4 thru 0x2dbfe7 (0x4944 bytes) 000371[0] 1743494403.168633 mapread 0x4f70d5 thru 0x4fe243 (0x716f bytes) 000372[1] 1743494403.265846 trunc from 0x9df9ca to 0x7d53a3 (0x20a626 bytes) 000373[0] 1743494403.386734 read 0x2d866f thru 0x2dccbf (0x4651 bytes) 000374[1] 1743494403.469994 read 0x766dae thru 0x768176 (0x13c9 bytes) 000376[0] 1743494403.554885 trunc from 0x7d53a4 to 0x38a042 (0x44b361 bytes) 000377[0] 1743494403.677583 read 0x2677dd thru 0x276419 (0xec3d bytes) 000379[1] 1743494404.004312 mapwrite 0x6833ec thru 0x68fcae (0xc8c3 bytes) 000380[0] 1743494404.421897 write 0x798547 thru 0x798b94 (0x64e bytes) 000384[0] 1743494404.608024 write 0x247c85 thru 0x255d44 (0xe0c0 bytes) 000386[0] 1743494404.746238 write 0x63c7dd thru 0x646c74 (0xa498 bytes) 000388[0] 1743494405.084154 trunc from 0x798b95 to 0x399265 (0x3ff92f bytes) 000390[1] 1743494405.213951 trunc from 0x399266 to 0x658d07 (0x2bfaa2 bytes) 000391[1] 1743494405.337176 read 0x3e3098 thru 0x3f05f5 (0xd55e bytes) 000395[0] 1743494405.398594 trunc from 0x658d08 to 0x781172 (0x12846b bytes) 000396[0] 1743494405.513047 read 0x09f67c thru 0x0ab8c4 (0xc249 bytes) 000398[0] 1743494405.547186 trunc from 0x781173 to 0x6220fa (0x15f078 bytes) 000399[0] 1743494405.623950 trunc from 0x6220fb to 0x2b8fd9 (0x369121 bytes) 000401[1] 1743494405.741455 mapwrite 0x197326 thru 0x19ffc0 (0x8c9b bytes) 000403[1] 1743494406.181274 read 0x0ee588 thru 0x0f7999 (0x9412 bytes) 000404[0] 1743494406.646620 mapread 0x257410 thru 0x25ba72 (0x4663 bytes) truncating to largest ever: 0x9e256c 000406[1] 1743494406.850544 trunc from 0x2b8fda to 0x9e256b (0x729592 bytes) 000411[1] 1743494407.045540 trunc from 0x9e256c to 0x84a5d0 (0x197f9b bytes) 000412[1] 1743494407.210415 mapread 0x1ed1c0 thru 0x1fab8b (0xd9cc bytes) 000414[1] 1743494407.697210 mapwrite 0x386482 thru 0x38b616 (0x5195 bytes) 000417[1] 1743494407.946776 mapread 0x0d54a2 thru 0x0e0b10 (0xb66f bytes) 000419[1] 1743494408.196714 read 0x804a15 thru 0x81370d (0xecf9 bytes) 000421[0] 1743494408.364908 write 0x8fa80c thru 0x8fc27a (0x1a6f bytes) 000422[1] 1743494408.438573 read 0x504ad6 thru 0x50910f (0x463a bytes) 000425[1] 1743494408.572791 mapread 0x684150 thru 0x68ff17 (0xbdc8 bytes) 000427[0] 1743494408.872752 trunc from 0x8fc27b to 0x389f89 (0x5722f1 bytes) 000430[1] 1743494409.095388 mapwrite 0x0df619 thru 0x0e742f (0x7e17 bytes) 000432[1] 1743494409.421290 trunc from 0x389f8a to 0x1a8f42 (0x1e1047 bytes) 000434[1] 1743494409.565602 mapwrite 0x15266a thru 0x154906 (0x229d bytes) 000438[1] 1743494409.903195 read 0x144e03 thru 0x151c7a (0xce78 bytes) 000441[0] 1743494410.087090 write 0x3792e4 thru 0x37b97f (0x269c bytes) 000442[0] 1743494410.237222 trunc from 0x37b980 to 0x1e01e5 (0x19b79a bytes) 000443[0] 1743494410.351448 read 0x07ad5a thru 0x085c70 (0xaf17 bytes) 000445[1] 1743494410.746562 mapwrite 0x999970 thru 0x99b6ee (0x1d7f bytes) 000447[0] 1743494411.676049 trunc from 0x99b6ef to 0x5e2f36 (0x3b87b8 bytes) 000451[0] 1743494411.946020 read 0x37e2a0 thru 0x387559 (0x92ba bytes) 000453[0] 1743494411.963594 mapread 0x41d3b8 thru 0x42bc04 (0xe84d bytes) 000454[0] 1743494412.014759 mapread 0x2830b4 thru 0x28779b (0x46e8 bytes) 000455[0] 1743494412.044341 mapread 0x033329 thru 0x03ba01 (0x86d9 bytes) 000458[0] 1743494412.171010 mapwrite 0x56f99e thru 0x570a85 (0x10e8 bytes) 000460[0] 1743494412.427926 mapread 0x32236c thru 0x32e051 (0xbce6 bytes) 000461[1] 1743494412.496913 write 0x0716ac thru 0x08040a (0xed5f bytes) 000465[0] 1743494412.774405 read 0x1bfe85 thru 0x1cba0a (0xbb86 bytes) 000466[0] 1743494412.849034 write 0x7bba9f thru 0x7c4aa8 (0x900a bytes) 000470[1] 1743494412.985976 write 0x1443fa thru 0x14b8c1 (0x74c8 bytes) 000471[0] 1743494413.076760 trunc from 0x7c4aa9 to 0x1d2f0d (0x5f1b9b bytes) 000472[1] 1743494413.169960 read 0x103ada thru 0x10a8a1 (0x6dc8 bytes) 000473[0] 1743494413.244867 trunc from 0x1d2f0e to 0x2c15da (0xee6cd bytes) 000474[1] 1743494413.362291 mapread 0x112ba5 thru 0x118a03 (0x5e5f bytes) 000475[1] 1743494413.442703 write 0x486740 thru 0x4944c3 (0xdd84 bytes) 000478[1] 1743494413.507198 mapread 0x4029f0 thru 0x40fec5 (0xd4d6 bytes) 000479[0] 1743494413.718111 mapwrite 0x6386ce thru 0x644afc (0xc42f bytes) 000481[1] 1743494414.110797 mapwrite 0x920984 thru 0x92e492 (0xdb0f bytes) 000487[1] 1743494414.629567 trunc from 0x92e493 to 0x28a7c8 (0x6a3cca bytes) 000489[0] 1743494414.769910 trunc from 0x28a7c9 to 0x132671 (0x158157 bytes) 000490[1] 1743494414.838518 trunc from 0x132672 to 0x8fd025 (0x7ca9b4 bytes) 000491[1] 1743494414.943395 mapwrite 0x8e5947 thru 0x8f5368 (0xfa22 bytes) 000494[0] 1743494415.335498 mapread 0x5634b1 thru 0x56a42f (0x6f7f bytes) 000496[1] 1743494415.479458 mapread 0x098147 thru 0x09945f (0x1319 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4415 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006[0] 1743494416.873276 write 0x5e6000 thru 0x5f082a (0xa82b bytes) 000007[0] 1743494416.902205 mapread 0x401000 thru 0x4109a8 (0xf9a9 bytes) truncating to largest ever: 0x84e1e4 000008[0] 1743494417.065681 trunc from 0x5f082b to 0x84e1e3 (0x25d9b9 bytes) truncating to largest ever: 0x9b6af7 000010[0] 1743494417.160434 trunc from 0x84e1e4 to 0x9b6af6 (0x168913 bytes) 000013[1] 1743494417.239555 mapwrite 0x08c000 thru 0x0986ac (0xc6ad bytes) 000014[1] 1743494417.609270 read 0x833000 thru 0x840d85 (0xdd86 bytes) 000015[1] 1743494417.636634 read 0x8aa000 thru 0x8aad5a (0xd5b bytes) 000018[0] 1743494417.707683 mapread 0x918000 thru 0x91b4c9 (0x34ca bytes) 000019[0] 1743494417.799254 read 0x083000 thru 0x0879dc (0x49dd bytes) 000020[0] 1743494417.832304 mapwrite 0x17c000 thru 0x17d520 (0x1521 bytes) 000021[1] 1743494418.042458 write 0x5d7000 thru 0x5dcaed (0x5aee bytes) 000022[1] 1743494418.078096 trunc from 0x9b6af7 to 0x76081e (0x2562d8 bytes) 000023[0] 1743494418.192558 read 0x07e000 thru 0x08b457 (0xd458 bytes) 000025[1] 1743494418.279136 mapwrite 0x164000 thru 0x170b1a (0xcb1b bytes) 000030[1] 1743494418.792130 mapread 0x4d9000 thru 0x4dc6bf (0x36c0 bytes) 000033[0] 1743494418.921584 read 0x2a2000 thru 0x2a6942 (0x4943 bytes) 000034[0] 1743494418.970561 trunc from 0x76081f to 0x8357ac (0xd4f8e bytes) 000035[1] 1743494419.030404 read 0x523000 thru 0x52e0f4 (0xb0f5 bytes) 000036[1] 1743494419.072274 read 0x691000 thru 0x6999b1 (0x89b2 bytes) 000037[1] 1743494419.109745 trunc from 0x8357ad to 0x567782 (0x2ce02a bytes) 000038[1] 1743494419.240130 mapwrite 0x373000 thru 0x380736 (0xd737 bytes) 000040[1] 1743494419.662096 trunc from 0x567783 to 0x348eb1 (0x21e8d1 bytes) 000041[1] 1743494419.737045 read 0x227000 thru 0x22f2ae (0x82af bytes) 000043[0] 1743494420.160740 read 0x2d3000 thru 0x2da781 (0x7782 bytes) 000044[1] 1743494420.510175 trunc from 0x348eb2 to 0x04873c (0x300775 bytes) 000045[1] 1743494420.597762 trunc from 0x04873d to 0x76d61e (0x724ee2 bytes) 000047[1] 1743494420.684927 mapwrite 0x546000 thru 0x54f5e7 (0x95e8 bytes) 000049[0] 1743494421.116254 write 0x6b6000 thru 0x6c0de8 (0xade9 bytes) 000050[0] 1743494421.493562 mapread 0x58a000 thru 0x593849 (0x984a bytes) 000051[1] 1743494421.521195 mapread 0x704000 thru 0x7124af (0xe4b0 bytes) 000052[1] 1743494421.759415 trunc from 0x76d61f to 0x0a0e3d (0x6cc7e1 bytes) 000053[1] 1743494421.841469 mapwrite 0x258000 thru 0x25995f (0x1960 bytes) 000054[0] 1743494422.054278 mapwrite 0x75a000 thru 0x764473 (0xa474 bytes) 000055[0] 1743494422.365144 mapread 0x305000 thru 0x3141aa (0xf1ab bytes) 000057[0] 1743494422.704713 write 0x057000 thru 0x05b018 (0x4019 bytes) 000058[1] 1743494422.716135 read 0x350000 thru 0x356a06 (0x6a07 bytes) 000059[1] 1743494422.809468 read 0x6ae000 thru 0x6bc1f3 (0xe1f4 bytes) 000061[1] 1743494422.832767 mapwrite 0x369000 thru 0x37318c (0xa18d bytes) 000065[0] 1743494423.098550 read 0x567000 thru 0x5728f5 (0xb8f6 bytes) 000066[1] 1743494423.147902 mapwrite 0x698000 thru 0x69b038 (0x3039 bytes) 000068[1] 1743494423.352340 trunc from 0x764474 to 0x725544 (0x3ef2f bytes) 000071[1] 1743494423.429950 read 0x45d000 thru 0x466066 (0x9067 bytes) 000072[0] 1743494423.450441 read 0x12e000 thru 0x135b31 (0x7b32 bytes) 000076[0] 1743494423.485560 write 0x467000 thru 0x470fa2 (0x9fa3 bytes) 000077[0] 1743494423.525573 trunc from 0x725545 to 0x929afa (0x2045b6 bytes) 000078[1] 1743494423.610027 trunc from 0x929afb to 0x00f3d9 (0x91a721 bytes) 000081[1] 1743494423.707750 mapwrite 0x6bc000 thru 0x6c3942 (0x7943 bytes) 000082[1] 1743494424.060993 mapwrite 0x36e000 thru 0x36e9b2 (0x9b3 bytes) 000083[0] 1743494424.182662 write 0x28e000 thru 0x2940a9 (0x60aa bytes) 000088[0] 1743494424.374003 mapwrite 0x309000 thru 0x309c42 (0xc43 bytes) 000089[1] 1743494424.588581 write 0x1b4000 thru 0x1c1bac (0xdbad bytes) 000090[1] 1743494424.715699 trunc from 0x6c3943 to 0x8403ca (0x17ca88 bytes) 000091[0] 1743494424.852805 write 0x804000 thru 0x80c972 (0x8973 bytes) 000093[1] 1743494425.032237 read 0x398000 thru 0x39cf7e (0x4f7f bytes) 000096[1] 1743494425.125957 trunc from 0x8403cb to 0x22df03 (0x6124c7 bytes) 000097[1] 1743494425.249019 read 0x008000 thru 0x011d28 (0x9d29 bytes) 000098[1] 1743494425.419905 read 0x0b3000 thru 0x0be5d4 (0xb5d5 bytes) 000099[1] 1743494425.451277 write 0x78f000 thru 0x791407 (0x2408 bytes) 000103[1] 1743494425.511520 mapwrite 0x8d2000 thru 0x8da3e0 (0x83e1 bytes) 000104[0] 1743494426.458739 mapread 0x640000 thru 0x64ab4a (0xab4b bytes) 000105[0] 1743494426.719659 mapwrite 0x7ad000 thru 0x7b6b93 (0x9b94 bytes) 000107[1] 1743494427.034065 write 0x674000 thru 0x6748f8 (0x8f9 bytes) 000108[1] 1743494427.095288 write 0x681000 thru 0x68b1bb (0xa1bc bytes) 000113[1] 1743494427.302283 mapread 0x0c3000 thru 0x0c87e1 (0x57e2 bytes) 000115[1] 1743494427.389109 trunc from 0x8da3e1 to 0x5fc21b (0x2de1c5 bytes) 000117[0] 1743494427.469324 read 0x580000 thru 0x5805a7 (0x5a8 bytes) 000118[0] 1743494427.527841 read 0x1c9000 thru 0x1cca08 (0x3a09 bytes) 000121[1] 1743494427.555780 mapwrite 0x971000 thru 0x97aad2 (0x9ad3 bytes) 000124[0] 1743494428.001509 trunc from 0x97aad3 to 0x6bf4b7 (0x2bb61b bytes) 000125[1] 1743494428.126267 read 0x5a7000 thru 0x5ace7f (0x5e80 bytes) 000127[1] 1743494428.200144 mapread 0x437000 thru 0x43f540 (0x8541 bytes) 000128[0] 1743494428.275475 trunc from 0x6bf4b8 to 0x5d6320 (0xe9197 bytes) 000132[0] 1743494428.404193 mapread 0x23a000 thru 0x241f80 (0x7f81 bytes) 000134[0] 1743494428.476923 mapwrite 0x1c8000 thru 0x1d5310 (0xd311 bytes) 000135[1] 1743494428.728827 mapwrite 0x791000 thru 0x79faa7 (0xeaa8 bytes) 000136[0] 1743494429.188516 mapwrite 0x307000 thru 0x310dfd (0x9dfe bytes) 000137[0] 1743494429.499836 read 0x419000 thru 0x4213f1 (0x83f2 bytes) 000138[0] 1743494429.517706 read 0x182000 thru 0x18bdb8 (0x9db9 bytes) 000141[1] 1743494429.640928 trunc from 0x79faa8 to 0x997887 (0x1f7de0 bytes) 000143[1] 1743494429.817955 read 0x36b000 thru 0x37a931 (0xf932 bytes) 000144[1] 1743494429.863680 mapread 0x6be000 thru 0x6c1a99 (0x3a9a bytes) 000146[0] 1743494429.947975 write 0x9c9000 thru 0x9c9053 (0x54 bytes) 000150[1] 1743494430.112226 read 0x2a4000 thru 0x2b1106 (0xd107 bytes) 000152[1] 1743494430.169262 write 0x283000 thru 0x2906cf (0xd6d0 bytes) 000153[1] 1743494430.224971 mapread 0x218000 thru 0x224aa0 (0xcaa1 bytes) 000154[1] 1743494430.293308 read 0x942000 thru 0x94ab0f (0x8b10 bytes) 000155[1] 1743494430.374028 trunc from 0x9c9054 to 0x821834 (0x1a781f bytes) 000156[0] 1743494430.482825 mapread 0x2a0000 thru 0x2a56c5 (0x56c6 bytes) 000157[0] 1743494430.615760 write 0x37c000 thru 0x3885e4 (0xc5e5 bytes) 000160[0] 1743494430.759430 mapread 0x159000 thru 0x15bb5a (0x2b5b bytes) 000161[0] 1743494431.019035 write 0x2ae000 thru 0x2b907f (0xb080 bytes) 000162[0] 1743494431.128792 write 0x00c000 thru 0x0132b1 (0x72b2 bytes) 000164[1] 1743494431.265332 write 0x3c3000 thru 0x3c577f (0x2780 bytes) 000165[0] 1743494431.378936 trunc from 0x821835 to 0x38355e (0x49e2d6 bytes) 000167[1] 1743494431.523481 trunc from 0x38355f to 0x98c0a4 (0x608b46 bytes) 000170[1] 1743494431.618976 read 0x72b000 thru 0x73047b (0x547c bytes) 000173[0] 1743494431.750623 read 0x3ff000 thru 0x40de73 (0xee74 bytes) 000179[1] 1743494431.815040 mapread 0x7d5000 thru 0x7d6fa3 (0x1fa4 bytes) 000180[1] 1743494431.841758 mapwrite 0x7d2000 thru 0x7db766 (0x9767 bytes) 000185[1] 1743494432.296744 write 0x92b000 thru 0x92b33c (0x33d bytes) 000186[1] 1743494432.352314 read 0x544000 thru 0x547271 (0x3272 bytes) 000187[0] 1743494432.376763 write 0x520000 thru 0x52f627 (0xf628 bytes) 000188[0] 1743494432.456217 trunc from 0x98c0a5 to 0x662cf0 (0x3293b4 bytes) 000189[0] 1743494432.542682 mapread 0x0c4000 thru 0x0c9709 (0x570a bytes) 000190[0] 1743494432.647482 mapwrite 0x559000 thru 0x5611dd (0x81de bytes) 000192[1] 1743494432.956659 read 0x42c000 thru 0x435b93 (0x9b94 bytes) 000193[0] 1743494432.998975 read 0x540000 thru 0x540ecb (0xecc bytes) 000195[0] 1743494433.012032 read 0x3d9000 thru 0x3e6c14 (0xdc15 bytes) 000197[1] 1743494433.029805 trunc from 0x662cf1 to 0x188a15 (0x4da2db bytes) 000200[1] 1743494433.116261 mapread 0x15d000 thru 0x1670a3 (0xa0a4 bytes) 000202[1] 1743494433.166549 read 0x02e000 thru 0x034a50 (0x6a51 bytes) 000203[1] 1743494433.372191 trunc from 0x188a16 to 0x097ecf (0xf0b46 bytes) 000207[1] 1743494433.593728 mapwrite 0x861000 thru 0x86875b (0x775c bytes) 000208[1] 1743494434.537187 mapread 0x7f8000 thru 0x804a57 (0xca58 bytes) 000209[0] 1743494434.538645 trunc from 0x86875c to 0x33f86e (0x528eed bytes) 000210[1] 1743494434.747613 mapread 0x004000 thru 0x00b516 (0x7517 bytes) 000211[1] 1743494434.903717 mapread 0x09b000 thru 0x09d2e3 (0x22e4 bytes) 000213[0] 1743494434.981167 read 0x16f000 thru 0x1752af (0x62b0 bytes) 000214[0] 1743494435.312012 mapwrite 0x950000 thru 0x954bf2 (0x4bf3 bytes) 000215[1] 1743494436.625826 write 0x6e5000 thru 0x6f029b (0xb29c bytes) 000216[1] 1743494436.794554 mapwrite 0x868000 thru 0x871e5b (0x9e5c bytes) 000219[1] 1743494437.240925 mapwrite 0x940000 thru 0x94e3b5 (0xe3b6 bytes) 000220[1] 1743494437.557555 write 0x930000 thru 0x9336ed (0x36ee bytes) 000222[1] 1743494437.559504 trunc from 0x954bf3 to 0x0e6dbd (0x86de35 bytes) 000223[0] 1743494437.649771 read 0x0ab000 thru 0x0b94af (0xe4b0 bytes) 000225[1] 1743494437.735933 trunc from 0x0e6dbe to 0x7af3e8 (0x6c862b bytes) 000226[0] 1743494437.821929 mapwrite 0x19a000 thru 0x19c6d1 (0x26d2 bytes) 000229[1] 1743494438.021239 read 0x12f000 thru 0x1370c2 (0x80c3 bytes) 000230[0] 1743494438.060607 mapwrite 0x9f0000 thru 0x9f1d57 (0x1d58 bytes) 000232[1] 1743494438.385443 mapwrite 0x21a000 thru 0x21d00f (0x3010 bytes) 000236[0] 1743494438.765869 read 0x1e0000 thru 0x1ee342 (0xe343 bytes) 000238[0] 1743494438.818169 mapwrite 0x75d000 thru 0x7611eb (0x41ec bytes) 000239[1] 1743494439.065921 mapread 0x160000 thru 0x16e461 (0xe462 bytes) 000240[1] 1743494439.188535 trunc from 0x9f1d58 to 0x7fa050 (0x1f7d07 bytes) 000243[1] 1743494439.411242 trunc from 0x7fa051 to 0x2285ab (0x5d1aa5 bytes) 000249[1] 1743494439.637922 mapwrite 0x4b6000 thru 0x4b9b35 (0x3b36 bytes) 000251[1] 1743494440.132916 read 0x20c000 thru 0x2125be (0x65bf bytes) 000252[0] 1743494440.228130 read_OD 0x3f4000 thru 0x3f9fff (0x6000 bytes) 000254[0] 1743494440.304240 mapread 0x2ea000 thru 0x2f1c7d (0x7c7e bytes) 000255[0] 1743494440.355331 mapwrite 0x9d8000 thru 0x9dee5c (0x6e5d bytes) 000257[0] 1743494440.768736 trunc from 0x9dee5d to 0x3cf64c (0x60f810 bytes) 000258[0] 1743494440.844196 mapwrite 0x8ca000 thru 0x8d4832 (0xa833 bytes) 000260[1] 1743494441.347676 read 0x24a000 thru 0x24d26f (0x3270 bytes) 000261[0] 1743494441.450600 write_OD 0x034000 thru 0x040fff (0xd000 bytes) 000264[0] 1743494441.673229 trunc from 0x8d4833 to 0x7efd98 (0xe4a9a bytes) 000266[1] 1743494441.792702 write 0x41b000 thru 0x421134 (0x6135 bytes) 000271[1] 1743494441.860025 trunc from 0x7efd99 to 0x1e8397 (0x607a01 bytes) 000272[0] 1743494441.931041 mapread 0x0dc000 thru 0x0dea8e (0x2a8f bytes) 000273[1] 1743494442.010193 read 0x1b0000 thru 0x1b36ad (0x36ae bytes) 000274[1] 1743494442.206713 mapread 0x168000 thru 0x16f263 (0x7264 bytes) 000276[0] 1743494442.209653 trunc from 0x1e8398 to 0x7fd554 (0x6151bd bytes) 000277[1] 1743494442.304607 write 0x801000 thru 0x8083c6 (0x73c7 bytes) 000278[0] 1743494442.360894 read_OD 0x44d000 thru 0x457fff (0xb000 bytes) 000280[1] 1743494442.452624 read 0x7fe000 thru 0x7ff962 (0x1963 bytes) 000281[0] 1743494442.493169 trunc from 0x8083c7 to 0x3b02ae (0x458118 bytes) 000282[1] 1743494442.580518 mapread 0x007000 thru 0x016de0 (0xfde1 bytes) 000284[0] 1743494442.668285 read_OD 00000000 thru 0x007fff (0x8000 bytes) 000285[0] 1743494442.687905 mapwrite 0x8ef000 thru 0x8f57f1 (0x67f2 bytes) 000288[1] 1743494443.744437 write 0x899000 thru 0x8a8ab3 (0xfab4 bytes) 000290[0] 1743494443.824646 mapwrite 0x8ba000 thru 0x8bfd7a (0x5d7b bytes) 000291[1] 1743494444.171183 trunc from 0x8f57f2 to 0x6ae3e5 (0x24740c bytes) truncating to largest ever: 0x9b9f7e 000292[1] 1743494444.286140 trunc from 0x6ae3e6 to 0x9b9f7d (0x30bb98 bytes) 000294[1] 1743494444.416162 trunc from 0x9b9f7e to 0x21cf4c (0x79d031 bytes) 000295[0] 1743494444.470432 mapread 0x0b2000 thru 0x0b7c27 (0x5c28 bytes) 000296[1] 1743494444.598529 write 0x75a000 thru 0x7605e7 (0x65e8 bytes) 000300[0] 1743494444.639780 read_OD 0x1fc000 thru 0x1fdfff (0x2000 bytes) 000301[1] 1743494444.689234 write 0x758000 thru 0x75baeb (0x3aec bytes) 000307[0] 1743494444.765941 mapwrite 0x59f000 thru 0x5ab8df (0xc8e0 bytes) 000311[1] 1743494445.622794 mapread 0x0bc000 thru 0x0bfe27 (0x3e28 bytes) 000312[0] 1743494445.729245 trunc from 0x7605e8 to 0x28cd6c (0x4d387b bytes) 000315[0] 1743494445.854158 mapwrite 0x34d000 thru 0x356e5c (0x9e5d bytes) 000317[1] 1743494446.340847 mapread 0x07e000 thru 0x080ffb (0x2ffc bytes) 000320[0] 1743494446.462609 mapwrite 0x3d2000 thru 0x3deacb (0xcacc bytes) 000321[0] 1743494446.878468 mapwrite 0x730000 thru 0x7317c3 (0x17c4 bytes) 000326[0] 1743494447.272701 write_OD 0x80a000 thru 0x80cfff (0x3000 bytes) 000328[0] 1743494447.433692 write_OD 0x5e1000 thru 0x5e1fff (0x1000 bytes) 000329[1] 1743494447.607000 read 0x2a9000 thru 0x2ab33e (0x233f bytes) 000332[0] 1743494447.629280 mapread 0x376000 thru 0x37cae3 (0x6ae4 bytes) 000334[1] 1743494447.727302 read 0x1e6000 thru 0x1f2f33 (0xcf34 bytes) 000336[1] 1743494447.746282 read 0x64d000 thru 0x65bec0 (0xeec1 bytes) 000341[0] 1743494447.765283 write_OD 0x355000 thru 0x355fff (0x1000 bytes) 000343[0] 1743494447.870962 trunc from 0x80d000 to 0x83d612 (0x30613 bytes) 000344[1] 1743494447.963279 mapread 0x554000 thru 0x55aec9 (0x6eca bytes) 000345[0] 1743494448.147122 write_OD 0x5ac000 thru 0x5b8fff (0xd000 bytes) 000346[0] 1743494448.369808 trunc from 0x83d613 to 0x49c57c (0x3a1096 bytes) 000349[1] 1743494448.521133 mapwrite 0x1f1000 thru 0x1fc0ad (0xb0ae bytes) 000352[0] 1743494448.880129 write_OD 0x92e000 thru 0x938fff (0xb000 bytes) 000354[0] 1743494449.017891 read_OD 0x6a2000 thru 0x6b0fff (0xf000 bytes) 000358[1] 1743494449.036932 mapread 0x632000 thru 0x63b519 (0x951a bytes) 000359[0] 1743494449.092386 read_OD 0x3c5000 thru 0x3d0fff (0xc000 bytes) 000364[0] 1743494449.113294 mapwrite 0x8d6000 thru 0x8e1528 (0xb529 bytes) 000366[1] 1743494449.433038 mapwrite 0x93e000 thru 0x9498d8 (0xb8d9 bytes) 000368[0] 1743494449.735202 read_OD 0x008000 thru 0x00efff (0x7000 bytes) 000369[1] 1743494449.793579 read 0x51e000 thru 0x527aac (0x9aad bytes) 000370[0] 1743494449.813017 mapwrite 0x885000 thru 0x891508 (0xc509 bytes) 000373[0] 1743494450.151907 write_OD 0x93f000 thru 0x947fff (0x9000 bytes) 000375[1] 1743494450.355133 read 0x6bd000 thru 0x6c9caf (0xccb0 bytes) 000381[1] 1743494450.417197 write 0x702000 thru 0x710892 (0xe893 bytes) 000385[1] 1743494450.525010 read 0x7df000 thru 0x7e0b3d (0x1b3e bytes) 000386[0] 1743494450.579777 read_OD 0x0ee000 thru 0x0effff (0x2000 bytes) 000387[1] 1743494450.636293 trunc from 0x9498d9 to 0x960d86 (0x174ae bytes) 000390[0] 1743494450.763620 write_OD 0x16f000 thru 0x17afff (0xc000 bytes) 000394[0] 1743494450.961515 mapwrite 0x6b1000 thru 0x6bc551 (0xb552 bytes) 000395[1] 1743494451.231555 mapwrite 0x862000 thru 0x866ded (0x4dee bytes) 000396[0] 1743494451.558487 mapread 0x36f000 thru 0x37c7b2 (0xd7b3 bytes) 000398[1] 1743494451.679168 read 0x1dc000 thru 0x1e90c1 (0xd0c2 bytes) 000400[0] 1743494451.703763 trunc from 0x960d87 to 0x92f87a (0x3150c bytes) 000402[1] 1743494451.787465 read 0x6a5000 thru 0x6b12d6 (0xc2d7 bytes) 000403[0] 1743494451.881847 mapwrite 0x4d5000 thru 0x4d9c97 (0x4c98 bytes) 000405[1] 1743494452.151847 read 0x45e000 thru 0x4683f1 (0xa3f2 bytes) 000408[0] 1743494452.225254 mapread 0x44d000 thru 0x455128 (0x8129 bytes) 000410[1] 1743494452.243897 trunc from 0x92f87b to 0x91b297 (0x145e3 bytes) 000415[1] 1743494452.436851 trunc from 0x91b298 to 0x8ab600 (0x6fc97 bytes) 000417[1] 1743494452.473338 mapread 0x694000 thru 0x6a2692 (0xe693 bytes) 000418[0] 1743494452.537379 mapread 0x4d0000 thru 0x4dc23e (0xc23f bytes) 000419[0] 1743494452.603464 mapwrite 0x406000 thru 0x4147e6 (0xe7e7 bytes) 000422[1] 1743494452.875042 trunc from 0x8ab601 to 0x7b9183 (0xf247d bytes) 000426[1] 1743494452.969820 mapwrite 0x4bc000 thru 0x4cade9 (0xedea bytes) 000428[1] 1743494453.198241 mapwrite 0x959000 thru 0x962292 (0x9293 bytes) 000429[1] 1743494453.411668 write 0x725000 thru 0x72f415 (0xa416 bytes) 000430[0] 1743494453.427555 read_OD 0x34c000 thru 0x351fff (0x6000 bytes) 000431[1] 1743494453.519199 mapwrite 0x385000 thru 0x38e6e8 (0x96e9 bytes) 000432[0] 1743494453.769962 trunc from 0x962293 to 0x7d8baf (0x1896e3 bytes) 000434[1] 1743494453.885331 write 0x6de000 thru 0x6e0d4f (0x2d50 bytes) 000437[0] 1743494454.010638 read_OD 0x6a4000 thru 0x6acfff (0x9000 bytes) 000439[1] 1743494454.067903 read 0x640000 thru 0x64b023 (0xb024 bytes) 000440[0] 1743494454.083095 read_OD 0x33e000 thru 0x33efff (0x1000 bytes) 000442[0] 1743494454.102754 write_OD 0x152000 thru 0x15bfff (0xa000 bytes) 000445[0] 1743494454.280411 write_OD 0x642000 thru 0x64bfff (0xa000 bytes) 000446[0] 1743494454.499463 trunc from 0x7d8bb0 to 0x6887b0 (0x1503ff bytes) 000448[0] 1743494454.628196 mapwrite 0x769000 thru 0x7715da (0x85db bytes) 000452[0] 1743494455.161004 write_OD 0x5a3000 thru 0x5aafff (0x8000 bytes) 000453[0] 1743494455.408119 trunc from 0x7715db to 0x264f37 (0x50c6a3 bytes) 000454[1] 1743494455.481677 trunc from 0x264f38 to 0x7e97ce (0x584897 bytes) 000458[0] 1743494455.616392 read_OD 0x16f000 thru 0x174fff (0x6000 bytes) 000461[0] 1743494455.662706 read_OD 0x7ae000 thru 0x7b5fff (0x8000 bytes) 000468[0] 1743494455.687703 mapread 0x394000 thru 0x399559 (0x555a bytes) 000472[1] 1743494455.720838 mapread 0x164000 thru 0x166b9e (0x2b9f bytes) 000473[1] 1743494455.769805 mapwrite 0x5d6000 thru 0x5e30e4 (0xd0e5 bytes) 000478[0] 1743494456.398575 write_OD 0x90f000 thru 0x90ffff (0x1000 bytes) 000479[1] 1743494456.668429 mapread 0x186000 thru 0x18736c (0x136d bytes) 000480[1] 1743494456.748417 mapread 0x195000 thru 0x19e1e0 (0x91e1 bytes) 000482[1] 1743494456.817016 mapread 0x6ce000 thru 0x6dd74d (0xf74e bytes) 000483[0] 1743494456.885615 trunc from 0x910000 to 0x6404fc (0x2cfb03 bytes) 000484[0] 1743494456.982759 mapwrite 0x513000 thru 0x517047 (0x4048 bytes) 000485[1] 1743494457.142802 write 0x18e000 thru 0x197b06 (0x9b07 bytes) 000486[1] 1743494457.222271 mapwrite 0x925000 thru 0x932420 (0xd421 bytes) 000489[1] 1743494457.713313 trunc from 0x932421 to 0x7b90cd (0x179353 bytes) 000492[1] 1743494457.876097 trunc from 0x7b90ce to 0x458fab (0x360122 bytes) 000494[0] 1743494457.972027 mapwrite 0x547000 thru 0x54bced (0x4cee bytes) 000496[0] 1743494458.357087 mapread 0x1cf000 thru 0x1d236f (0x3370 bytes) 000498[1] 1743494458.518270 mapread 0x32e000 thru 0x33d1f6 (0xf1f7 bytes) 000500[0] 1743494458.708991 mapwrite 0x59a000 thru 0x5a6666 (0xc667 bytes) All operations completed A-OK! PASS 16a (121s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:01:13 (1743494473) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.66277 s, 5.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.13748 s, 7.4 MB/s Chance of close/open is 1 in 50 Seed set to 4481 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000014[0] 1743494483.782940 mapwrite 0x18bc54 thru 0x18c9ae (0xd5b bytes) 000016[1] 1743494484.193564 mapwrite 0x5ea28c thru 0x5eb79c (0x1511 bytes) 000017[0] 1743494484.790092 mapread 0x5b25b6 thru 0x5b2ce6 (0x731 bytes) 000018[0] 1743494484.917853 read 0x3bc27d thru 0x3bd45e (0x11e2 bytes) truncating to largest ever: 0x775725 000022[1] 1743494484.964417 trunc from 0x5eb79d to 0x775724 (0x189f88 bytes) 000023[0] 1743494485.123263 trunc from 0x775725 to 0x6409ea (0x134d3a bytes) 000024[1] 1743494485.252791 mapread 0x328b6f thru 0x328e4a (0x2dc bytes) 000027[1] 1743494485.475050 mapread 0x331d9b thru 0x3328b2 (0xb18 bytes) 000028[1] 1743494485.577551 write 0x902f15 thru 0x9049db (0x1ac7 bytes) 000037[1] 1743494486.048558 mapwrite 0x6e901e thru 0x6eaf61 (0x1f44 bytes) 000038[1] 1743494486.356375 mapread 0x322144 thru 0x322f37 (0xdf4 bytes) 000039[1] 1743494486.380434 mapwrite 0x68946c thru 0x68a265 (0xdfa bytes) 000041[1] 1743494486.698481 mapwrite 0x30b409 thru 0x30d186 (0x1d7e bytes) 000042[0] 1743494487.067762 read 0x7bd7f6 thru 0x7bf3a9 (0x1bb4 bytes) 000043[0] 1743494487.173504 write 0x057d1e thru 0x058d68 (0x104b bytes) 000044[0] 1743494487.286061 mapread 0x714673 thru 0x714ec1 (0x84f bytes) 000048[0] 1743494487.309763 mapread 0x0a05f2 thru 0x0a0642 (0x51 bytes) 000050[1] 1743494487.416171 mapread 0x1090c6 thru 0x1096eb (0x626 bytes) 000054[1] 1743494487.496933 read 0x17db30 thru 0x17f99b (0x1e6c bytes) 000057[0] 1743494487.511613 mapwrite 0x92b3b8 thru 0x92cd4d (0x1996 bytes) 000059[1] 1743494487.844127 read 0x07f2ce thru 0x080073 (0xda6 bytes) truncating to largest ever: 0x91bbc6 000060[0] 1743494487.895333 trunc from 0x92cd4e to 0x91bbc5 (0x11188 bytes) 000061[0] 1743494487.981820 read 0x34b23d thru 0x34b86b (0x62f bytes) 000062[1] 1743494487.998517 trunc from 0x91bbc6 to 0x6db752 (0x240473 bytes) 000063[0] 1743494488.146536 trunc from 0x6db753 to 0x194169 (0x5475e9 bytes) 000064[0] 1743494488.277859 trunc from 0x19416a to 0x3eb6cb (0x257562 bytes) 000067[1] 1743494488.404102 trunc from 0x3eb6cc to 0x03ffbf (0x3ab70c bytes) 000071[0] 1743494488.760437 mapread 0x0260f6 thru 0x027a16 (0x1921 bytes) 000073[1] 1743494488.899881 write 0x413306 thru 0x41344d (0x148 bytes) 000077[1] 1743494488.960327 read 0x244132 thru 0x244914 (0x7e3 bytes) 000079[1] 1743494488.969208 write 0x6e1df1 thru 0x6e3940 (0x1b50 bytes) 000081[0] 1743494489.009712 write 0x2b2cc3 thru 0x2b48f8 (0x1c36 bytes) 000082[0] 1743494489.120662 read 0x6a73b3 thru 0x6a7754 (0x3a2 bytes) 000084[1] 1743494489.291662 read 0x022c4b thru 0x024165 (0x151b bytes) 000085[1] 1743494489.379074 write 0x94d2e5 thru 0x94de4b (0xb67 bytes) 000086[0] 1743494489.426438 trunc from 0x94de4c to 0x2e99fa (0x664451 bytes) 000087[1] 1743494489.680842 mapwrite 0x00a8c8 thru 0x00be88 (0x15c1 bytes) 000089[0] 1743494490.254565 write 0x4f8f5a thru 0x4f9c09 (0xcb0 bytes) 000090[0] 1743494490.277429 read 0x158a9d thru 0x158d82 (0x2e6 bytes) 000091[0] 1743494490.309845 mapwrite 0x2c4f98 thru 0x2c6b49 (0x1bb2 bytes) 000092[1] 1743494490.511496 write 0x9f133c thru 0x9f17c2 (0x487 bytes) 000093[1] 1743494490.576914 trunc from 0x9f17c3 to 0x86101f (0x1907a3 bytes) 000096[1] 1743494490.696245 mapwrite 0x75e464 thru 0x75fa3f (0x15dc bytes) 000098[1] 1743494490.834688 trunc from 0x861020 to 0x0931be (0x7cde61 bytes) 000099[1] 1743494490.885760 read 0x063d07 thru 0x064e4c (0x1146 bytes) 000100[0] 1743494490.992914 mapread 0x07ab8b thru 0x07c4c4 (0x193a bytes) 000104[0] 1743494491.248104 read 0x02709c thru 0x028b08 (0x1a6d bytes) 000106[0] 1743494491.365031 mapwrite 0x7dbcfc thru 0x7dd0af (0x13b4 bytes) 000110[1] 1743494492.231934 write 0x64a599 thru 0x64b5e9 (0x1051 bytes) 000111[1] 1743494492.332433 read 0x124561 thru 0x124e18 (0x8b8 bytes) 000112[1] 1743494492.374515 trunc from 0x7dd0b0 to 0x1fc7dc (0x5e08d3 bytes) 000114[1] 1743494492.471477 mapread 0x0cdbd1 thru 0x0cea80 (0xeb0 bytes) 000115[1] 1743494492.542096 mapread 0x00960e thru 0x0096c7 (0xba bytes) 000116[0] 1743494492.566184 write 0x7a59d4 thru 0x7a69e8 (0x1015 bytes) 000118[0] 1743494492.646010 write 0x9b8ce9 thru 0x9b9632 (0x94a bytes) 000119[0] 1743494492.672865 read 0x7979bf thru 0x798fd7 (0x1619 bytes) 000120[0] 1743494492.684813 mapread 0x2858fe thru 0x2866eb (0xdee bytes) 000124[0] 1743494492.792252 mapread 0x3341f6 thru 0x335b16 (0x1921 bytes) 000128[0] 1743494492.847320 write 0x0db7c7 thru 0x0dcfd4 (0x180e bytes) 000129[0] 1743494492.861896 mapread 0x32a323 thru 0x32c078 (0x1d56 bytes) 000133[1] 1743494493.012027 mapread 0x842a2b thru 0x8439d0 (0xfa6 bytes) 000135[1] 1743494493.096607 write 0x95148c thru 0x9515c0 (0x135 bytes) 000138[0] 1743494493.195299 trunc from 0x9b9633 to 0x2017be (0x7b7e74 bytes) 000141[0] 1743494493.372697 read 0x041767 thru 0x042ad2 (0x136c bytes) 000142[1] 1743494493.631533 read 0x1259d8 thru 0x126b2a (0x1153 bytes) 000143[1] 1743494494.024053 mapread 0x021ba7 thru 0x021e07 (0x261 bytes) 000144[0] 1743494494.093511 mapread 0x1d58b1 thru 0x1d7077 (0x17c7 bytes) 000148[0] 1743494494.167015 trunc from 0x2017bf to 0x319903 (0x118145 bytes) 000151[0] 1743494494.357646 read 0x080cdc thru 0x082aca (0x1def bytes) truncating to largest ever: 0x9f9ae4 000154[1] 1743494494.780029 trunc from 0x319904 to 0x9f9ae3 (0x6e01e0 bytes) 000155[1] 1743494494.925669 trunc from 0x9f9ae4 to 0x394a18 (0x6650cb bytes) 000156[1] 1743494495.010758 write 0x74e5ab thru 0x750002 (0x1a58 bytes) 000157[0] 1743494495.055684 read 0x5fcc01 thru 0x5feb6f (0x1f6f bytes) 000159[0] 1743494495.168196 mapwrite 0x4779f2 thru 0x4790e0 (0x16ef bytes) 000160[0] 1743494495.359982 trunc from 0x750003 to 0x83c168 (0xec166 bytes) 000161[0] 1743494495.438905 mapwrite 0x0d9c2b thru 0x0dad72 (0x1148 bytes) 000163[1] 1743494495.701804 mapwrite 0x802779 thru 0x803199 (0xa21 bytes) 000165[1] 1743494496.213229 trunc from 0x83c169 to 0x2bdbd5 (0x57e593 bytes) 000166[1] 1743494496.310593 mapwrite 0x840789 thru 0x840a45 (0x2bd bytes) 000170[0] 1743494496.952732 mapread 0x5e2c2f thru 0x5e4bcc (0x1f9e bytes) 000173[1] 1743494497.167844 read 0x829d7a thru 0x82ad54 (0xfdb bytes) 000174[0] 1743494497.216482 trunc from 0x840a46 to 0x4cbfbb (0x374a8a bytes) 000175[1] 1743494497.406744 write 0x22c00d thru 0x22d3c5 (0x13b9 bytes) 000178[1] 1743494497.613673 read 0x08d7fd thru 0x08d827 (0x2b bytes) 000181[1] 1743494497.742555 mapread 0x1fd8be thru 0x1fda25 (0x168 bytes) 000184[0] 1743494497.868224 mapread 0x05bb67 thru 0x05cf24 (0x13be bytes) 000185[1] 1743494498.016462 write 0x5dd9e7 thru 0x5df30b (0x1925 bytes) 000186[0] 1743494498.041460 mapwrite 0x71c004 thru 0x71c1bd (0x1ba bytes) 000188[0] 1743494498.378589 trunc from 0x71c1be to 0x4dfa7a (0x23c743 bytes) 000189[1] 1743494498.411131 mapwrite 0x2fa4dd thru 0x2fb043 (0xb67 bytes) 000191[1] 1743494498.675974 mapwrite 0x68ba98 thru 0x68d055 (0x15be bytes) 000192[0] 1743494499.083591 write 0x846905 thru 0x84868b (0x1d87 bytes) 000196[0] 1743494499.235840 mapwrite 0x6f2ad7 thru 0x6f414c (0x1676 bytes) 000197[1] 1743494499.463196 mapwrite 0x534fe7 thru 0x5362ee (0x1308 bytes) 000199[0] 1743494499.868538 write 0x57d474 thru 0x57d956 (0x4e3 bytes) 000204[1] 1743494500.029280 write 0x78e1c6 thru 0x790164 (0x1f9f bytes) 000206[1] 1743494500.178507 read 0x61dd54 thru 0x61f3cc (0x1679 bytes) 000209[1] 1743494500.267082 mapread 0x6d85d7 thru 0x6d98d2 (0x12fc bytes) 000211[1] 1743494500.347020 mapread 0x2b6696 thru 0x2b6dec (0x757 bytes) 000214[1] 1743494500.386118 mapread 0x470e0d thru 0x471ab7 (0xcab bytes) 000215[1] 1743494500.442683 mapwrite 0x6132cb thru 0x613bdc (0x912 bytes) 000216[0] 1743494500.643286 trunc from 0x84868c to 0x1790b5 (0x6cf5d6 bytes) 000217[0] 1743494500.780820 mapwrite 0x233320 thru 0x234931 (0x1612 bytes) 000218[0] 1743494500.999280 write 0x4e0025 thru 0x4e0a80 (0xa5c bytes) 000219[1] 1743494501.086973 read 0x1e414e thru 0x1e4800 (0x6b3 bytes) 000220[0] 1743494501.156862 mapread 0x21f46a thru 0x220bca (0x1761 bytes) 000222[0] 1743494501.190095 mapread 0x3248ab thru 0x3253b3 (0xb09 bytes) 000228[0] 1743494501.352588 mapread 0x01d632 thru 0x01df12 (0x8e1 bytes) 000236[1] 1743494501.518685 write 0x7313d6 thru 0x7331c6 (0x1df1 bytes) 000239[1] 1743494501.630663 mapread 0x5991ea thru 0x5999b7 (0x7ce bytes) 000243[1] 1743494501.716195 trunc from 0x7331c7 to 0x2a2bab (0x49061b bytes) 000245[1] 1743494501.838517 mapwrite 0x4cf2e5 thru 0x4d06e7 (0x1403 bytes) 000246[0] 1743494502.061396 read 0x1c4b86 thru 0x1c4e74 (0x2ef bytes) 000247[1] 1743494502.476312 trunc from 0x4d06e8 to 0x08c26f (0x444478 bytes) 000248[0] 1743494502.637610 mapwrite 0x5a0d1f thru 0x5a23a7 (0x1689 bytes) 000249[0] 1743494503.019434 mapread 0x524741 thru 0x525ebb (0x177b bytes) 000250[1] 1743494503.035225 trunc from 0x5a23a8 to 0x679715 (0xd736e bytes) 000252[1] 1743494503.239300 trunc from 0x679716 to 0x1777f1 (0x501f24 bytes) 000253[0] 1743494503.279169 mapwrite 0x983ff0 thru 0x985f3a (0x1f4b bytes) 000254[0] 1743494504.068846 read 0x8b41c9 thru 0x8b4622 (0x45a bytes) 000256[1] 1743494504.084517 read 0x51dd52 thru 0x51ef90 (0x123f bytes) 000257[0] 1743494504.176920 write 0x5068cb thru 0x5086d9 (0x1e0f bytes) 000261[1] 1743494504.297263 read 0x8bb8b1 thru 0x8bc600 (0xd50 bytes) 000262[1] 1743494504.365270 read 0x5a6221 thru 0x5a70fd (0xedd bytes) 000265[0] 1743494504.427898 read 0x1a8f62 thru 0x1aaa4f (0x1aee bytes) 000266[1] 1743494504.442319 mapwrite 0x8d4c6b thru 0x8d573f (0xad5 bytes) 000268[0] 1743494504.733785 mapwrite 0x2a1e19 thru 0x2a3c3c (0x1e24 bytes) 000269[1] 1743494504.969778 trunc from 0x985f3b to 0x561dac (0x42418e bytes) 000270[0] 1743494505.083229 mapread 0x329eb0 thru 0x32a20a (0x35b bytes) 000272[1] 1743494505.177437 trunc from 0x561dad to 0x065b64 (0x4fc248 bytes) 000274[1] 1743494505.238094 write 0x80fef9 thru 0x8105ed (0x6f5 bytes) 000275[0] 1743494505.288956 trunc from 0x8105ee to 0x4243af (0x3ec23e bytes) 000278[0] 1743494505.524895 mapread 0x14a201 thru 0x14a812 (0x612 bytes) 000279[0] 1743494505.544390 mapwrite 0x505318 thru 0x50647b (0x1164 bytes) 000281[0] 1743494505.720688 read 0x068573 thru 0x06a3d3 (0x1e61 bytes) 000288[1] 1743494505.999534 mapread 0x43e68d thru 0x43ea49 (0x3bd bytes) 000289[1] 1743494506.068528 write 0x7db243 thru 0x7dc04b (0xe09 bytes) 000293[1] 1743494506.186740 trunc from 0x7dc04c to 0x6edf8f (0xee0bc bytes) 000295[0] 1743494506.251340 trunc from 0x6edf90 to 0x822877 (0x1348e8 bytes) 000299[1] 1743494506.560259 trunc from 0x822878 to 0x73d98f (0xe4ee8 bytes) 000300[0] 1743494506.731445 mapwrite 0x490345 thru 0x490e82 (0xb3e bytes) 000303[1] 1743494507.009430 write 0x1365cd thru 0x136acd (0x501 bytes) 000307[0] 1743494507.116922 trunc from 0x73d990 to 0x0c447c (0x679513 bytes) 000308[1] 1743494507.221873 mapread 0x01f492 thru 0x02026d (0xddc bytes) 000309[0] 1743494507.360308 write 0x12e2db thru 0x12eed5 (0xbfb bytes) 000310[0] 1743494507.388500 mapread 0x06b463 thru 0x06c9da (0x1578 bytes) 000315[0] 1743494507.492981 mapwrite 0x1df916 thru 0x1e1524 (0x1c0f bytes) 000316[0] 1743494507.781884 trunc from 0x1e1525 to 0x0effb4 (0xf1570 bytes) 000318[1] 1743494507.910436 mapread 0x05d0b6 thru 0x05d150 (0x9b bytes) 000319[1] 1743494507.999042 write 0x97dc5d thru 0x97f07e (0x1422 bytes) 000322[0] 1743494508.111773 mapread 0x8b6564 thru 0x8b6b3b (0x5d8 bytes) 000323[0] 1743494508.187068 read 0x46a5e6 thru 0x46c526 (0x1f41 bytes) 000324[1] 1743494508.204863 write 0x10234d thru 0x102414 (0xc8 bytes) 000327[1] 1743494508.327942 write 0x16ba7d thru 0x16bcbe (0x242 bytes) 000330[0] 1743494508.519834 trunc from 0x97f07f to 0x5c81c2 (0x3b6ebc bytes) 000331[0] 1743494508.650434 mapwrite 0x61d0db thru 0x61d36b (0x291 bytes) 000334[1] 1743494509.040657 mapread 0x27acb0 thru 0x27bddd (0x112e bytes) 000335[1] 1743494509.169479 read 0x1e0629 thru 0x1e1f8f (0x1967 bytes) 000336[0] 1743494509.197981 trunc from 0x61d36c to 0x56214a (0xbb221 bytes) 000337[1] 1743494509.325549 trunc from 0x56214b to 0x3c6921 (0x19b829 bytes) 000338[1] 1743494509.499730 mapread 0x3a3f69 thru 0x3a4d9a (0xe32 bytes) 000339[0] 1743494509.588105 mapwrite 0x6bff0a thru 0x6c1a44 (0x1b3b bytes) 000341[0] 1743494509.939140 mapread 0x0244c3 thru 0x026082 (0x1bc0 bytes) 000344[1] 1743494510.158865 mapread 0x6533ee thru 0x654486 (0x1099 bytes) 000345[0] 1743494510.246488 trunc from 0x6c1a45 to 0x899ab6 (0x1d8072 bytes) 000347[0] 1743494510.407792 trunc from 0x899ab7 to 0x348a6f (0x551047 bytes) 000349[0] 1743494510.545184 write 0x0289b0 thru 0x02a6fc (0x1d4d bytes) 000352[1] 1743494510.597899 mapread 0x1e3066 thru 0x1e4893 (0x182e bytes) 000353[1] 1743494510.740980 mapread 0x19b22e thru 0x19cdff (0x1bd2 bytes) 000356[0] 1743494510.823732 write 0x309d7e thru 0x309f17 (0x19a bytes) 000357[1] 1743494510.862024 write 0x7ede14 thru 0x7ef0d5 (0x12c2 bytes) 000361[0] 1743494510.994603 read 0x4508b0 thru 0x45220b (0x195c bytes) 000362[0] 1743494511.112197 write 0x6529f7 thru 0x652a0b (0x15 bytes) 000364[1] 1743494511.171844 read 0x7923a1 thru 0x792716 (0x376 bytes) 000366[1] 1743494511.265003 read 0x4da1d6 thru 0x4dc14b (0x1f76 bytes) 000367[1] 1743494511.281861 trunc from 0x7ef0d6 to 0x69996a (0x15576b bytes) 000368[0] 1743494511.417651 write 0x970ba1 thru 0x97204d (0x14ad bytes) 000371[0] 1743494511.559221 trunc from 0x97204e to 0x21d8d0 (0x75477d bytes) 000372[0] 1743494511.721178 mapread 0x03a7b9 thru 0x03aa85 (0x2cd bytes) 000374[1] 1743494511.787617 read 0x04fc7c thru 0x050402 (0x787 bytes) 000378[1] 1743494512.070826 write 0x52ae9e thru 0x52c31f (0x1482 bytes) 000379[1] 1743494512.126967 mapread 0x229265 thru 0x229fcf (0xd6b bytes) 000380[0] 1743494512.134579 mapread 0x175c9f thru 0x176da4 (0x1106 bytes) 000381[1] 1743494512.180678 mapread 0x528b9c thru 0x529f63 (0x13c8 bytes) 000384[1] 1743494512.199105 read 0x1a2ea3 thru 0x1a338c (0x4ea bytes) 000386[1] 1743494512.202449 mapwrite 0x32189e thru 0x322480 (0xbe3 bytes) 000387[1] 1743494512.453156 mapwrite 0x05f4f6 thru 0x0604a7 (0xfb2 bytes) 000389[0] 1743494512.618525 read 0x4ea330 thru 0x4ea486 (0x157 bytes) 000394[1] 1743494512.715126 trunc from 0x52c320 to 0x24efba (0x2dd365 bytes) 000395[1] 1743494512.791669 mapwrite 0x20f4a7 thru 0x20fb0f (0x669 bytes) 000400[0] 1743494513.052636 mapwrite 0x3b6696 thru 0x3b7378 (0xce3 bytes) 000403[0] 1743494513.381291 mapwrite 0x3e91b1 thru 0x3e9601 (0x451 bytes) 000404[1] 1743494513.663861 mapread 0x03f982 thru 0x0400a6 (0x725 bytes) 000409[1] 1743494513.919038 write 0x77454d thru 0x776269 (0x1d1d bytes) 000410[1] 1743494513.981687 read 0x65d5e9 thru 0x65ec84 (0x169c bytes) 000411[0] 1743494514.049303 read 0x6087e5 thru 0x60a518 (0x1d34 bytes) 000412[0] 1743494514.154299 trunc from 0x77626a to 0x7d9b7c (0x63913 bytes) 000413[1] 1743494514.304651 read 0x5b3b1e thru 0x5b4857 (0xd3a bytes) 000414[0] 1743494514.349177 write 0x227e85 thru 0x229304 (0x1480 bytes) 000417[0] 1743494514.365122 read 0x11b854 thru 0x11d04c (0x17f9 bytes) 000418[0] 1743494514.373020 write 0x37de62 thru 0x37e455 (0x5f4 bytes) 000419[1] 1743494514.409621 write 0x8c5bc8 thru 0x8c5ebc (0x2f5 bytes) 000422[0] 1743494514.491286 read 0x77815a thru 0x77892e (0x7d5 bytes) 000423[0] 1743494514.521627 write 0x4d9181 thru 0x4d9f14 (0xd94 bytes) 000425[0] 1743494514.624959 mapread 0x80988f thru 0x80a600 (0xd72 bytes) 000426[1] 1743494514.699701 write 0x6237ee thru 0x62546c (0x1c7f bytes) 000427[0] 1743494514.816827 trunc from 0x8c5ebd to 0x3a6f9e (0x51ef1e bytes) 000428[0] 1743494514.934322 trunc from 0x3a6f9f to 0x6d3b1c (0x32cb7e bytes) 000429[0] 1743494514.998197 mapread 0x39244b thru 0x393f42 (0x1af8 bytes) 000430[1] 1743494515.032476 mapread 0x5bc9f7 thru 0x5be784 (0x1d8e bytes) 000434[1] 1743494515.184855 write 0x20666c thru 0x207ffd (0x1992 bytes) 000435[1] 1743494515.307966 mapwrite 0x54ac6c thru 0x54c203 (0x1598 bytes) 000436[1] 1743494515.543596 trunc from 0x6d3b1d to 0x21b4c2 (0x4b865a bytes) 000437[0] 1743494515.726857 trunc from 0x21b4c3 to 0x617c14 (0x3fc752 bytes) 000439[1] 1743494515.935861 mapread 0x30134a thru 0x301715 (0x3cc bytes) 000446[1] 1743494516.151982 trunc from 0x617c15 to 0x088075 (0x58fb9f bytes) 000447[1] 1743494516.301814 mapwrite 0x23e7cf thru 0x23ece6 (0x518 bytes) 000449[0] 1743494516.586244 read 0x054192 thru 0x05426e (0xdd bytes) 000451[1] 1743494516.915529 read 0x080467 thru 0x0816dd (0x1277 bytes) 000452[0] 1743494517.275049 read 0x1fbbcd thru 0x1fc996 (0xdca bytes) 000453[0] 1743494517.275908 mapwrite 0x13af2a thru 0x13af88 (0x5f bytes) 000454[0] 1743494517.850940 mapwrite 0x906b2e thru 0x907749 (0xc1c bytes) 000455[0] 1743494518.740951 trunc from 0x90774a to 0x857dca (0xaf97f bytes) 000458[0] 1743494518.934526 trunc from 0x857dcb to 0x388f67 (0x4cee63 bytes) 000459[1] 1743494519.076487 mapread 0x088bf6 thru 0x08a343 (0x174e bytes) 000464[0] 1743494519.648288 write 0x2a6199 thru 0x2a6922 (0x78a bytes) 000467[1] 1743494519.804201 trunc from 0x388f68 to 0x7b3943 (0x42a9dc bytes) 000469[0] 1743494520.016963 trunc from 0x7b3944 to 0x22b32e (0x588615 bytes) 000470[0] 1743494520.089574 mapread 0x0d0f5f thru 0x0d2a35 (0x1ad7 bytes) 000475[0] 1743494520.206504 trunc from 0x22b32f to 0x7c119a (0x595e6c bytes) 000477[1] 1743494520.351238 mapread 0x5e53ee thru 0x5e5a15 (0x628 bytes) 000478[1] 1743494520.445573 mapread 0x2c7a85 thru 0x2c95b2 (0x1b2e bytes) 000480[1] 1743494520.600099 mapread 0x0c63c9 thru 0x0c7fa6 (0x1bde bytes) 000484[1] 1743494520.619574 mapwrite 0x8d4863 thru 0x8d4869 (0x7 bytes) 000486[0] 1743494521.169949 mapwrite 0x1e61a8 thru 0x1e6833 (0x68c bytes) 000488[0] 1743494521.406940 mapread 0x1b6eee thru 0x1b77ac (0x8bf bytes) 000489[1] 1743494521.512454 mapwrite 0x27c28a thru 0x27d5a6 (0x131d bytes) 000491[0] 1743494521.777942 trunc from 0x8d486a to 0x57096b (0x363efe bytes) 000493[0] 1743494521.948327 trunc from 0x57096c to 0x294746 (0x2dc225 bytes) 000494[0] 1743494522.037039 trunc from 0x294747 to 0x43f54e (0x1aae08 bytes) 000496[1] 1743494522.116020 mapread 0x0f8c3c thru 0x0f990f (0xcd4 bytes) 000497[1] 1743494522.207872 read 0x0cf2e4 thru 0x0d124e (0x1f6b bytes) 000499[0] 1743494522.265059 mapread 0x2b4516 thru 0x2b61bf (0x1caa bytes) 000500[0] 1743494522.345403 mapread 0x0f81c1 thru 0x0f9e6c (0x1cac bytes) All operations completed A-OK! PASS 16b (64s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:02:18 (1743494538) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (9s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:02:27 (1743494547) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.68517 s, 5.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.869068 s, 9.7 MB/s ldlm.namespaces.MGC192.168.206.111@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f0086354000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9f0086554000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9f0086554000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9f0086554000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 26.8122 s, 3.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.24478 s, 16.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.60002 s, 22.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9138 s, 8.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 21.0433 s, 5.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.10518 s, 11.5 MB/s PASS 16d (113s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:04:21 (1743494661) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.645202 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280957 s, 3.7 MB/s PASS 16e (18s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:04:38 (1743494678) TIMEOUT PASS 16f (43s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:05:21 (1743494721) TIMEOUT PASS 16g (41s) == sanityn test 16h: mmap read after truncate file ======= 04:06:03 (1743494763) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (18s) == sanityn test 16i: read after truncate file ============ 04:06:21 (1743494781) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (20s) == sanityn test 16j: race dio with buffered i/o ========== 04:06:42 (1743494802) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68282 s, 7.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19951 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.69746 s, 4.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.71994 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.00269 s, 3.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.6387 s, 16.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.53108 s, 4.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.02023 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.90455 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.45164 s, 3.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 8.04246 s, 3.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99567 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.82093 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.26279 s, 3.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.86167 s, 3.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95178 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.76529 s, 7.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.85648 s, 3.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 8.19919 s, 3.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (91s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:08:13 (1743494893) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Apr 1 04:10:09 EDT 2025 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Tue Apr 1 04:10:10 EDT 2025 + fsx 3 done: rc=0 Tue Apr 1 04:10:10 EDT 2025 + fsx 4 done: rc=0 Tue Apr 1 04:10:10 EDT 2025 + fsx 5 done: rc=0 Tue Apr 1 04:10:10 EDT 2025 PASS 16k (141s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:10:35 (1743495035) fail_loc=0x8000030a PASS 17 (23s) == sanityn test 18: mmap sanity check =========================================================================================== 04:10:57 (1743495057) mmap test1: basic mmap operation (PASS, 0.25888s) mmap test2: MAP_PRIVATE not write back (PASS, 0.18865s) mmap test3: concurrent mmap ops on two nodes (PASS, 20.337s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 5.0158s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.33699s) mmap test6: check mmap write/read content on two nodes (PASS, 0.43928s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.4379s) mmap test11: multi-thread shared mmap (PASS, 2.3333s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1655s) PASS 18 (53s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:11:50 (1743495110) oleg611-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (11s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:12:01 (1743495121) PASS 20 (17s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:12:18 (1743495138) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (17s) == sanityn test 23: others should see updated atime while another read============================================================== 04:12:35 (1743495155) now is 1743495162 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 new atime is 1743495224 PASS 23 (83s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:13:59 (1743495239) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (18s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:14:16 (1743495256) PASS 24b (17s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:14:33 (1743495273) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (18s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:14:51 (1743495291) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (8s) == sanityn test 26a: allow mtime to get older ============ 04:15:00 (1743495300) utime: good mknod times 1743495302 <= 1743495303 <= 1743495303 for /mnt/lustre/f26a utime: good mknod times 1743495302 <= 1743495303 <= 1743495303 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (17s) == sanityn test 26b: sync mtime between ost and mds ====== 04:15:17 (1743495317) PASS 26b (18s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 04:15:35 (1743495335) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-04-01 04:15:39.000000000 -0400 Birth: 2025-04-01 04:15:39.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-04-01 04:15:39.000000000 -0400 Birth: 2025-04-01 04:15:39.000000000 -0400 PASS 26c (16s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 04:15:59 (1743495359) 0 PASS 30 (19s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:16:18 (1743495378) fail_loc=0x314 PASS 31a (14s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:16:33 (1743495393) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg611-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.111@tcp:/lustre 7542784 40960 7495680 1% /mnt/lustre2 PASS 31b (40s) == sanityn test 31r: open-rename(replace) race =========== 04:17:12 (1743495432) -rw-r--r-- 1 root root 0 Apr 1 04:17 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (20s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 04:17:33 (1743495453) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1213: 42935 Terminated tail -f $tfile PASS 31s (17s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 04:17:50 (1743495470) PASS 31t (17s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:18:11 (1743495491) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (8s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 04:18:19 (1743495499) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (7s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 04:18:27 (1743495507) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (7s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 04:18:34 (1743495514) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (8s) == sanityn test 34: no lock timeout under IO ============= 04:18:43 (1743495523) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid in FULL state after 0 sec oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid in IDLE state after 0 sec oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid in FULL state after 0 sec oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid in IDLE state after 0 sec oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0086354000.ost_server_uuid in FULL state after 0 sec oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0086354000.ost_server_uuid in IDLE state after 0 sec PASS 34 (200s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 04:22:03 (1743495723) fail_loc=0x80000317 Race attempt 0 Wait for 49086 49100 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1752: 49086 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.111@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.206.111@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 - open/close 308 (time 1743495748.06 total 10.01 last 30.78) - open/close 635 (time 1743495758.08 total 20.02 last 32.65) - open/close 980 (time 1743495768.08 total 30.02 last 34.49) - open/close 1342 (time 1743495778.10 total 40.05 last 36.11) - open/close 1668 (time 1743495788.11 total 50.05 last 32.59) - open/close 1999 (time 1743495798.12 total 60.06 last 33.07) - open/close 2265 (time 1743495808.13 total 70.08 last 26.56) - open/close 2584 (time 1743495818.13 total 80.08 last 31.90) - open/close 2979 (time 1743495828.14 total 90.09 last 39.47) - open/close 3372 (time 1743495838.16 total 100.11 last 39.20) - open/close 3722 (time 1743495848.17 total 110.12 last 34.97) total: 4000 open/close in 117.37 seconds: 34.08 ops/second PASS 35 (133s) == sanityn test 36: handle ESTALE/open-unlink correctly == 04:24:15 (1743495855) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg611-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.2228 s, 23.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7438336) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.23674 s, 23.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7441408) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.12906 s, 24.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.43733 s, 21.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82193 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77162 s, 29.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.31221 s, 22.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06036 s, 25.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.55436 s, 20.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.79397 s, 18.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.49568 s, 21.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7443456) after(7496704) PASS 36 (479s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 04:32:15 (1743496335) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 - create 734 (time 1743496353.65 total 10.00 last 73.38) - create 1392 (time 1743496363.66 total 20.01 last 65.73) - create 2009 (time 1743496373.67 total 30.03 last 61.61) - create 2683 (time 1743496383.68 total 40.03 last 67.39) - create 3335 (time 1743496393.68 total 50.03 last 65.17) - create 3982 (time 1743496403.69 total 60.04 last 64.67) - create 4599 (time 1743496413.69 total 70.04 last 61.69) - create 5313 (time 1743496423.69 total 80.04 last 71.39) - create 5917 (time 1743496433.69 total 90.05 last 60.38) - create 6539 (time 1743496443.70 total 100.06 last 62.14) - create 7187 (time 1743496453.76 total 110.12 last 64.39) - create 7893 (time 1743496463.81 total 120.16 last 70.32) - create 8577 (time 1743496473.81 total 130.17 last 68.34) - create 9228 (time 1743496483.82 total 140.18 last 65.04) - create 9887 (time 1743496493.88 total 150.23 last 65.55) total: 10000 create in 152.60 seconds: 65.53 ops/second PASS 37 (180s) == sanityn test 39a: file mtime does not change after rename ========================================================== 04:35:14 (1743496514) repeat after cancel_lru_locks PASS 39a (18s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 04:35:32 (1743496532) repeat after cancel_lru_locks PASS 39b (17s) == sanityn test 39c: check truncate mtime update ================================================================================ 04:35:50 (1743496550) repeat after cancel_lru_locks PASS 39c (15s) == sanityn test 39d: sync write should update mtime ====== 04:36:06 (1743496566) fail_loc=0x411 fail_loc=0 PASS 39d (15s) == sanityn test 40a: pdirops: create vs others ======================================================================== 04:36:21 (1743496581) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (37s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 04:36:58 (1743496618) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (35s) == sanityn test 40c: pdirops: link and others ======================================================================== 04:37:34 (1743496654) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (37s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 04:38:11 (1743496691) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (38s) == sanityn test 40e: pdirops: rename and others ======================================================================== 04:38:49 (1743496729) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (38s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 04:39:27 (1743496767) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (35s) == sanityn test 41b: pdirops: create vs create ======================================================================== 04:40:02 (1743496802) open: File exists Conflict PASS 41b (30s) == sanityn test 41c: pdirops: create vs link ======================================================================== 04:40:33 (1743496833) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (32s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 04:41:04 (1743496864) Conflict PASS 41d (34s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 04:41:38 (1743496898) Conflict PASS 41e (31s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 04:42:10 (1743496930) Conflict PASS 41f (33s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 04:42:42 (1743496962) Conflict PASS 41g (28s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 04:43:11 (1743496991) Conflict PASS 41h (34s) == sanityn test 41i: reint_open: create vs create ======== 04:43:45 (1743497025) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1683s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 05:11:49 (1743498709) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (21s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 05:12:10 (1743498730) open: File exists Conflict PASS 42b (21s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 05:12:31 (1743498751) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (21s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 05:12:52 (1743498772) Conflict PASS 42d (21s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 05:13:13 (1743498793) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (21s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 05:13:34 (1743498814) Conflict PASS 42f (20s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 05:13:55 (1743498835) Conflict PASS 42g (19s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 05:14:14 (1743498854) Conflict PASS 42h (21s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 05:14:36 (1743498876) PASS 43a (100s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 05:16:16 (1743498976) Conflict PASS 43b (20s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 05:16:37 (1743498997) Conflict PASS 43c (20s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 05:16:57 (1743499017) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (21s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 05:17:18 (1743499038) Conflict PASS 43e (21s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 05:17:40 (1743499060) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (22s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 05:18:02 (1743499082) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (22s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 05:18:23 (1743499103) Conflict PASS 43h (21s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 05:18:44 (1743499124) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (6s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 05:18:50 (1743499130) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (212s) == sanityn test 43k: unlink vs create ==================== 05:22:23 (1743499343) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1634s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 05:49:38 (1743500978) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Apr 1 05:49:49 EDT 2025 PASS 44a (19s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 05:49:56 (1743500996) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 05:50:14 (1743501014) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (18s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 05:50:32 (1743501032) Conflict PASS 44d (18s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 05:50:50 (1743501050) Conflict PASS 44e (18s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 05:51:08 (1743501068) Conflict PASS 44f (18s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 05:51:26 (1743501086) Conflict PASS 44g (17s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 05:51:43 (1743501103) Conflict PASS 44h (19s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 05:52:02 (1743501122) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 05:52:07 (1743501127) PASS 45a (117s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 05:54:04 (1743501244) Conflict PASS 45b (17s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 05:54:21 (1743501261) Conflict PASS 45c (18s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 05:54:39 (1743501279) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 05:54:57 (1743501297) Conflict PASS 45e (19s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 05:55:15 (1743501315) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (18s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 05:55:33 (1743501333) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (18s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 05:55:51 (1743501351) Conflict PASS 45h (19s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 05:56:10 (1743501370) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 05:56:14 (1743501374) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1332s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:18:26 (1743502706) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 06:18:40 (1743502720) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 06:18:54 (1743502734) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:19:08 (1743502748) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 06:19:22 (1743502762) Conflict PASS 46e (15s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 06:19:37 (1743502777) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 06:19:52 (1743502792) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 06:20:06 (1743502806) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 06:20:20 (1743502820) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:20:24 (1743502824) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:20:27 (1743502827) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:20:31 (1743502831) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:20:34 (1743502834) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:20:37 (1743502837) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:20:41 (1743502841) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:20:44 (1743502844) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:20:48 (1743502848) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0718164 s, 143 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:21:00 (1743502860) 0+1 records in 0+1 records out 158 bytes copied, 0.00886219 s, 17.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3519: kill: (260896) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:21:09 (1743502869) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0115121 s, 88.9 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239361 s, 43.8 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:21:29 (1743502889) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0115031 s, 89.0 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:21:42 (1743502902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368516 s, 28.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:21:50 (1743502910) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2449 0x991 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2449 0x991 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 06:22:00 (1743502920) llite.lustre-ffff9f0086354000.xattr_cache=1 llite.lustre-ffff9f0086554000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505009 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:01.000000000 -0400 Modify: 2025-04-01 06:22:02.000000000 -0400 Change: 2025-04-01 06:22:02.000000000 -0400 Birth: 2025-04-01 06:22:02.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:07.000000000 -0400 Modify: 2025-04-01 06:22:08.000000000 -0400 Change: 2025-04-01 06:22:08.000000000 -0400 Birth: 2025-04-01 06:22:08.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:14.000000000 -0400 Modify: 2025-04-01 06:22:14.000000000 -0400 Change: 2025-04-01 06:22:14.000000000 -0400 Birth: 2025-04-01 06:22:14.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:13.000000000 -0400 Modify: 2025-04-01 06:22:14.000000000 -0400 Change: 2025-04-01 06:22:14.000000000 -0400 Birth: 2025-04-01 06:22:14.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:20.000000000 -0400 Modify: 2025-04-01 06:22:21.000000000 -0400 Change: 2025-04-01 06:22:21.000000000 -0400 Birth: 2025-04-01 06:22:20.000000000 -0400 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:22:20.000000000 -0400 Modify: 2025-04-01 06:22:21.000000000 -0400 Change: 2025-04-01 06:22:21.000000000 -0400 Birth: 2025-04-01 06:22:20.000000000 -0400 PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 06:22:31 (1743502951) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 06:22:44 (1743502964) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 06:22:57 (1743502977) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 06:23:17 (1743502997) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 06:23:32 (1743503012) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1743503014 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.924834 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (75s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 06:24:47 (1743503087) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1743503101 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.17379 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (129s) == sanityn test 60: Verify data_version behaviour ======== 06:26:57 (1743503217) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 06:27:08 (1743503228) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:27:18 (1743503238) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:27:28 (1743503248) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00422108 s, 15.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:27:32 (1743503252) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00749103 s, 8.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:27:36 (1743503256) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:27:39 (1743503259) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.013524 s, 4.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:27:43 (1743503263) llite.lustre-ffff9f0086354000.xattr_cache=1 llite.lustre-ffff9f0086554000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:27:50 (1743503270) llite.lustre-ffff9f0086354000.xattr_cache=1 llite.lustre-ffff9f0086554000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:27:58 (1743503278) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00995888 s, 103 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00524231 s, 195 kB/s 272189: get lock2 272191: get lock1 272191: done 272191: get lock2 272189: done 272189: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 272191: cannot get lock2: Resource deadlock avoided 272191: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 272189: cannot get lock1: Resource deadlock avoided 272189: done 272189: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:28:09 (1743503289) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0509607 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145586 s, 72.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4108: 272773 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 06:28:21 (1743503301) opening files: ........ closing files: ........ PASS 76 (46s) == sanityn test 77a: check FIFO NRS policy =============== 06:29:07 (1743503347) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180405 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127274 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162961 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199413 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223805 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164024 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244533 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102555 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126638 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118386 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241009 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319295 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.534371 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272379 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350654 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501683 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208768 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899554 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194016 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172942 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568048 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101655 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051923 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13858 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121559 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757791 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518499 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757577 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989339 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581761 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325359 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258482 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310062 s, 33.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 06:29:18 (1743503358) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164292 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114802 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139291 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403543 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.545515 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466467 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483594 s, 2.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567529 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.570471 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565995 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444753 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.602632 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441198 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49621 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.659969 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523776 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.712406 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533323 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781373 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683284 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481642 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962252 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588553 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298995 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542769 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185153 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012823 s, 81.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038479 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370987 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311529 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013365 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119638 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125696 s, 83.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179508 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718671 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871081 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.685934 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.622951 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.6964 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.67351 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.670243 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.716055 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648532 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.700622 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.701794 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.717461 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.739717 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.718786 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.726041 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.781411 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737048 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993167 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908169 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174749 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164398 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136494 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167753 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774909 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11505 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188056 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526299 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954966 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012837 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113707 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120953 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161603 s, 64.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (17s) == sanityn test 77c: check ORR NRS policy ================ 06:29:36 (1743503376) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183702 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705639 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992789 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.649681 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.781644 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.733092 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.78949 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.778771 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.68557 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.689424 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.800078 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.702063 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.741419 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.727491 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.757914 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.829124 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.851377 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629722 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760871 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783768 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957926 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115446 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306459 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713194 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542439 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404591 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524847 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349055 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349413 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114834 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130036 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120224 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337728 s, 31.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166365 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620765 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046493 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532943 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603044 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321912 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325635 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5017 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355483 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447373 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407672 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37982 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.567115 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463374 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500934 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42068 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565112 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114081 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557404 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483177 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978675 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640876 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435882 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614753 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561416 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116569 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590573 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699888 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615069 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679063 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902489 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693051 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758497 s, 13.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195224 s, 85.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375019 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910821 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759633 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482275 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548584 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.679428 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.577486 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.69802 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.590413 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.593293 s, 1.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.750675 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.824318 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.816029 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.829256 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.717104 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.820284 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142008 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237523 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248312 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188115 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213938 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112407 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142905 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1424 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856326 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029928 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681724 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120733 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748259 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762346 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027718 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571577 s, 18.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (19s) == sanityn test 77d: check TRR nrs policy ================ 06:29:54 (1743503394) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.192629 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190788 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203494 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.724026 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.780327 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.691943 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648444 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.719002 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.775815 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.864239 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.586271 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.605466 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.773145 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.857676 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.675265 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.607728 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.833595 s, 1.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112909 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114154 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412325 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503461 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823108 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605992 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796512 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828371 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120946 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711796 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033381 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142897 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322989 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022589 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152142 s, 68.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167301 s, 62.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179466 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756996 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527328 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720454 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11311 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124928 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326367 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340467 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148079 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127143 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562038 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132449 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196922 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170687 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219255 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31818 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340599 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280368 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656072 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367687 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459236 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357194 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103305 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115825 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12215 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348014 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107347 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117261 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10406 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380192 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310614 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531235 s, 19.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687481 s, 153 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.211056 s, 79.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123947 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107732 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173379 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211191 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.625919 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.632773 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.770708 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.807686 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.726003 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.631628 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.768438 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.659426 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.813992 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.671874 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.628362 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.788084 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958491 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791326 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134961 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08608 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100012 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111505 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720101 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144628 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114244 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702212 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800732 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121949 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966789 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134464 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949809 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236192 s, 44.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 06:30:11 (1743503411) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.11@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.189281 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100842 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899417 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245135 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225663 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229071 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14954 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167958 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230017 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293781 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297732 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201559 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170892 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298954 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232611 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267963 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254038 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688562 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190863 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106028 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157415 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206114 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207098 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147866 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198655 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181056 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117764 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14704 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176899 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208594 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134908 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150109 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219827 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=5 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150386 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624329 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969621 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119823 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105527 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211882 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109885 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123286 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178107 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25084 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260579 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160609 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146308 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193294 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230132 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3086 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257534 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106407 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523865 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098755 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547717 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704075 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599392 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658155 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105772 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121773 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702386 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075822 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134402 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307878 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151102 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125048 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153932 s, 68.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201248 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725344 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784864 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859299 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131227 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131118 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125712 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109341 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128361 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114338 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928558 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116682 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17446 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240169 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173981 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273991 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407381 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136011 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869544 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109384 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229463 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721054 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110047 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114721 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122974 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11119 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107567 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842447 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872596 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334215 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184765 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554969 s, 18.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211419 s, 49.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.191625 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934893 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10896 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893903 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468442 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486973 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494667 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.620282 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.573295 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529478 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.856531 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.851178 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.933144 s, 1.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.833547 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.940905 s, 1.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.878702 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.954273 s, 1.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484927 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522584 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957506 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484297 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914425 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571666 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812484 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824647 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121138 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089602 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326248 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267966 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211209 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264529 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191576 s, 54.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117731 s, 89.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (23s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:30:34 (1743503434) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193016 s, 86.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787129 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118821 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134568 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18382 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143376 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102842 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171427 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145143 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985379 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139072 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139838 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203806 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251175 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363471 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309546 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283968 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02333 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722207 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564137 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345059 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787791 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362676 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215919 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544865 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121113 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328078 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348642 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125224 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327528 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423692 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265174 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160048 s, 65.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166622 s, 101 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142679 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164557 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138228 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175983 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311807 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136522 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214059 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210108 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238694 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420895 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383495 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268524 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374532 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161629 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24848 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203244 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490604 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788788 s, 13.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555113 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783995 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289584 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123329 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107251 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110204 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131958 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995046 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150838 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105156 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553286 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297188 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208862 s, 50.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280412 s, 37.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.18179 s, 92.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15325 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198416 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209196 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232443 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309782 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419339 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399505 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462028 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718089 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216758 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407208 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276335 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29056 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392058 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234443 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275561 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913561 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429108 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247808 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451946 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892987 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127215 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630809 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238293 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566509 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366579 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012579 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340035 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136692 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147059 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049834 s, 21.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.189808 s, 88.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430383 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819016 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957372 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215366 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206893 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232531 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186374 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293705 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326101 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336096 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392542 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390204 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400859 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483075 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511062 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440038 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475231 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353364 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031242 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460422 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551762 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415769 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773036 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651894 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446204 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250312 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115386 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312595 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127278 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127972 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144959 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137885 s, 76.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (22s) == sanityn test 77g: Change TBF type directly ============ 06:30:56 (1743503456) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:31:07 (1743503467) oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg611-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg611-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 12 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg611-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 12 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg611-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 12 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 06:31:18 (1743503478) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:31:39 (1743503499) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371325 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729382 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930749 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916225 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159771 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180333 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151751 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116804 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121463 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114644 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16434 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258506 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159116 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175574 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263417 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278847 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256785 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393797 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392591 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695656 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878358 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364243 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865281 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619842 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707273 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363629 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126336 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330883 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125256 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014413 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220244 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137416 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103405 s, 101 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.67302 s, 15.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4602 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (64s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:32:43 (1743503563) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23283 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958887 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102263 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107607 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148864 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134051 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231747 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295865 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217065 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31008 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188276 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299876 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242625 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226239 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278642 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358896 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322139 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124679 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850633 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072785 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132812 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158438 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366346 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712915 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160092 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18361 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181194 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654427 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024616 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062647 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227955 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01938 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396988 s, 264 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7979 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8354 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23789 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719325 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080821 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820033 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907615 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176557 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146792 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897962 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846896 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119455 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198936 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196574 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273822 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106736 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11172 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126238 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509517 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740848 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896238 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931743 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760897 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805676 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948915 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109822 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643536 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900667 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760346 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884694 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124097 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187331 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128385 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179432 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348681 s, 30.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8622 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8393 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 06:35:16 (1743503716) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23552 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131264 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119458 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142789 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164163 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152582 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230585 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244018 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891269 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153223 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200184 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789763 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173854 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235791 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114545 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239804 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35133 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756604 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780304 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891019 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594209 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648586 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893381 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100544 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741645 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565848 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465887 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234838 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122092 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234855 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233598 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105182 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175433 s, 59.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1556 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8337 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24199 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12257 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120635 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141983 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135129 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147332 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150094 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205378 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174121 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224463 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269834 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272241 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322975 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292431 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432336 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336153 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293448 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609772 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039994 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637126 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623533 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542881 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327578 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071503 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256088 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252605 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252132 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321499 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313056 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136481 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116099 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126506 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128195 s, 81.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1872 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8388 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (150s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:37:46 (1743503866) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.419077 s, 40.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688702 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471241 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879734 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195423 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203454 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194925 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150222 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780906 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215197 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139425 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194152 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151175 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16912 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290681 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205851 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235505 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744306 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137146 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135173 s, 77.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457115 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864268 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137003 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167253 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013341 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697736 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332936 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106403 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710872 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407938 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204357 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634346 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369746 s, 28.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.88773 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75728 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273488 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782304 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721302 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185202 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105315 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19557 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135133 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174046 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586794 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252511 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175261 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240367 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195232 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154019 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136994 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147357 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186831 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795743 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443668 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703832 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255783 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888407 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557567 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746579 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645537 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034455 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180942 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235071 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135446 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368012 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147853 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137545 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017748 s, 59.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.53863 s, 18.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75738 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300249 s, 55.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070943 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948756 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981896 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154996 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079442 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169251 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176326 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126812 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164159 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156676 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260192 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216907 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166318 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191881 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23395 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18027 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240165 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264991 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330578 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127013 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103684 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535582 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978783 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849676 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614953 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897034 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656532 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133043 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275007 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075408 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102763 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00265179 s, 395 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.94113 s, 17.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90323 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467284 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992078 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103047 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794444 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109393 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110011 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114376 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11778 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219279 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124945 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190125 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232982 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231519 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190201 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272534 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275806 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292143 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255584 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411006 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508455 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584797 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468321 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813254 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670269 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693371 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763436 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334515 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732762 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136012 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911974 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820391 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00454202 s, 231 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99148 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75632 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46739 s, 19.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76706 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.987308 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143776 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246681 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185802 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202579 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117343 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111037 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101069 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120957 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239046 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396076 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258282 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430981 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317322 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248787 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140368 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455905 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017243 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150242 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151485 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770535 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364801 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213834 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271619 s, 38.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134059 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200037 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141539 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02859 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078661 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138373 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136077 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019625 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141729 s, 74.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6612 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4582 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.442696 s, 37.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888212 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101969 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11766 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142236 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291349 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239102 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29128 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351367 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247546 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197358 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214737 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260625 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233756 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309086 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411719 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299134 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924047 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602304 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886391 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880206 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415702 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099931 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723745 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345917 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106976 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767112 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165686 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354037 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220866 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325484 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193351 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739727 s, 142 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.45404 s, 16.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77253 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.553081 s, 30.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242141 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185959 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185456 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292434 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305913 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431866 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.570649 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390101 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23721 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.597633 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207191 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472865 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19152 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202936 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438612 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531936 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068284 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741077 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106393 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109886 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146407 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117576 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092056 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1358 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528095 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839505 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375233 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266527 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383088 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235571 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352049 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161089 s, 65.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg611-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg611-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9412 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76216 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0133 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76778 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (406s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:44:32 (1743504272) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:44:43 (1743504283) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg611-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.39247 s, 309 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg611-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0604 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:45:42 (1743504342) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.365902 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925347 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108039 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146295 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125925 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174406 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151621 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109163 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15737 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141821 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27409 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198621 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26803 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103583 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255962 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817129 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336532 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9f0086354000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994748 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125325 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886832 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174642 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118398 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120733 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159984 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825077 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178637 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828646 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126759 s, 82.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809364 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994308 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149145 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189709 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187629 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9f0086354000.lock_unused_count=3 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.2009 s, 16.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90952 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.351404 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547263 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686084 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670591 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113155 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103156 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126294 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19889 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188659 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20559 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852581 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140472 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422154 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566512 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122696 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189918 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229297 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438101 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643511 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325752 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740684 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096729 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104031 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775692 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598752 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621504 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523962 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186559 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317965 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129633 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186334 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142943 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134883 s, 77.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg611-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg611-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.26729 s, 16.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91669 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (79s) == sanityn test 77o: Changing rank should not panic ====== 06:47:02 (1743504422) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:47:13 (1743504433) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (149s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:49:42 (1743504582) mds.MDS.mdt.nrs_policies=tbf oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg611-client: oleg611-server: ssh exited with exit code 36 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 oleg611-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg611-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg611-client: oleg611-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (30s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 06:50:12 (1743504612) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 299060 kill md_thread xargs PASS 77r (74s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:51:26 (1743504686) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 06:51:37 (1743504697) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505967 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:51:38.000000000 -0400 Modify: 2025-04-01 06:51:38.000000000 -0400 Change: 2025-04-01 06:51:39.000000000 -0400 Birth: 2025-04-01 06:51:38.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:51:55 (1743504715) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:51:58 (1743504718) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:52:02 (1743504722) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:52:05 (1743504725) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:52:09 (1743504729) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 06:52:12 (1743504732) - open/close 1564 (time 1743504747.62 total 10.00 last 156.38) total: 2500 open/close in 16.46 seconds: 151.90 ops/second snapshot_time 1743504819.832536187 secs.nsecs start_time 1743504735.685635221 secs.nsecs elapsed_time 84.146900966 secs.nsecs open 2505 samples [usecs] 281 29642 4589765 13199737615 close 2504 samples [usecs] 247 20033 1445297 1476882581 mknod 2501 samples [usecs] 805 29263 4178984 11156553190 unlink 2500 samples [usecs] 1093 181628 7351668 133174232108 mkdir 2 samples [usecs] 4270 6269 10539 57533261 rmdir 2 samples [usecs] 4943 6857 11800 71451698 rename 2500 samples [usecs] 914 77404 15932510 223544114924 getattr 3761 samples [usecs] 8 2155 128591 11321323 parallel_rename_file 2500 samples [usecs] 914 77404 15932510 223544114924 crossdir_rename 2500 samples [usecs] 914 77404 15932510 223544114924 PASS 81d (93s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 06:53:45 (1743504825) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7365 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7365 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 06:53:52 (1743504832) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 06:53:55 (1743504835) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 06:54:09 (1743504849) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1743504851 Finishing test test1 at 1743504854 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 06:54:20 (1743504860) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 06:54:24 (1743504864) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 06:54:28 (1743504868) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 06:54:32 (1743504872) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5365 0x14f5 0x280000400 0 5346 0x14e2 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5366 0x14f6 0x280000400 0 5347 0x14e3 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 06:54:49 (1743504889) osc.lustre-OST0000-osc-ffff9f0086354000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9f0086554000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f0086354000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9f0086554000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.149515 s, 54.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.06302 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5744: 314434 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0479035 s, 85.5 kB/s osc.lustre-OST0000-osc-ffff9f0086354000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9f0086554000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0086354000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0086554000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 06:55:06 (1743504906) llite.lustre-ffff9f0086354000.fast_read=0 llite.lustre-ffff9f0086554000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0258491 s, 81.1 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9f0086354000.fast_read=1 llite.lustre-ffff9f0086554000.fast_read=1 PASS 95a (26s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 06:55:31 (1743504931) llite.lustre-ffff9f0086354000.fast_read=0 llite.lustre-ffff9f0086554000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00395499 s, 3.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9f0086354000.fast_read=1 llite.lustre-ffff9f0086554000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 06:55:45 (1743504945) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 06:55:48 (1743504948) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00325118 s, 1.3 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 06:55:55 (1743504955) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0307889 s, 68.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 06:56:02 (1743504962) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0470994 s, 44.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 06:56:10 (1743504970) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00390669 s, 3.1 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 06:56:18 (1743504978) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00237783 s, 1.7 MB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 06:56:25 (1743504985) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00293378 s, 1.4 MB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 06:56:31 (1743504991) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00305017 s, 1.3 MB/s PASS 101c (9s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 06:56:41 (1743505001) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1743505004 lockahead_test: lockahead_test.c:1191: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1743505007 Finishing test test23 at 1743505007 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 06:56:55 (1743505015) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 06:56:58 (1743505018) fail_loc=0x416 PASS 105 (33s) == sanityn test 106a: Verify the btime via statx() ======= 06:57:31 (1743505051) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 06:57:35 (1743505055) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00590045 s, 174 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508502 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:57:37.000000000 -0400 Modify: 2025-04-01 06:57:37.000000000 -0400 Change: 2025-04-01 06:57:37.000000000 -0400 Birth: 2025-04-01 06:57:37.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272508502 -rw-r--r-- 1 0 0 1743505057 1743505057 1743505057 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508502 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 06:57:37.000000000 -0400 Modify: 2025-04-01 06:57:37.000000000 -0400 Change: 2025-04-01 06:57:37.000000000 -0400 Birth: 2025-04-01 06:57:37.000000000 -0400 1743505057 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 06:57:44 (1743505064) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 06:57:52 (1743505072) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307345 s, 34.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5355 0x14eb 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 06:58:03 (1743505083) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.27524 s, 38.1 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5373 0x14fd 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7365 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 06:58:18 (1743505098) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00378354 s, 2.6 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.95189 s, 1.7 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 06:58:30 (1743505110) 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre 192.168.206.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 192.168.206.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project 0 0 Stopping client oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg611-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre2 PASS 109 (1947s) == sanityn test 110: do not grant another lock on resend ========================================================== 07:30:58 (1743507058) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 07:31:00.000000000 -0400 Modify: 2025-04-01 07:31:00.000000000 -0400 Change: 2025-04-01 07:31:00.000000000 -0400 Birth: 2025-04-01 07:31:00.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg611-client.virtnet oleg611-client.virtnet PASS 110 (71s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 07:32:09 (1743507129) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 07:32:13 (1743507133) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (5s) == sanityn test 113: check servers of specified fs ======= 07:32:18 (1743507138) lustre-OST0000-osc-ffff9f00918ec000 active. lustre-OST0001-osc-ffff9f00918ec000 active. lustre-OST0000-osc-ffff9f00918ec000 active. lustre-OST0001-osc-ffff9f00918ec000 active. MGC192.168.206.111@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 07:32:27 (1743507147) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 07:32:31 (1743507151) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (5s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 07:32:36 (1743507156) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 07:32:40 (1743507160) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=22 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1836 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1907 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2198 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2329 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2004 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2035 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.229 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2278 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.209 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.9609 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.6417 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.9336 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1281 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0946 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.9958 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2454 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 39.6973 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0632 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1891 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1905 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 59.652 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 59.9628 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1173 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3841 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 59.9854 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2356 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.028 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.0651 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.8357 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.8471 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.8395 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.8245 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.1353 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.8683 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 79.9187 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.4049 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.1513 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 99.8666 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 99.9425 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 99.9618 s, 0.0 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 99.9571 s, 0.0 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 100.057 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.292 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.088 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f00918e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f00918e9000.ost_server_uuid in IDLE state after 0 sec PASS 200 (122s) cleanup: ====================================================== == sanityn test complete, duration 13788 sec ============= 07:34:44 (1743507284) === sanityn: start cleanup 07:34:47 (1743507287) === Stopping clients: oleg611-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg611-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 07:37:10 (1743507430) ===