-----============= acceptance-small: sanityn ============----- Tue Nov 18 19:04:11 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 19:04:26 (1763510666) === Starting client oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Mount client oleg138-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Started clients oleg138-client.virtnet: 192.168.201.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9af785e88800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af798626000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af785e88800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af798626000.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 19:04:50 (1763510690) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402324 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7595] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 19:04:52 (1763510692) PASS 0 (6s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 19:04:58 (1763510698) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 19:05:04 (1763510704) -rw-r--r-- 1 root root 0 Nov 18 19:05 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 19:05:09 (1763510709) -rw-r--r-- 1 root root 0 Nov 18 19:05 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 19:05:13 (1763510713) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (4s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 19:05:17 (1763510717) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 19:05:22 (1763510722) total 1 -rwxrwxrwx 1 root root 0 Nov 18 19:05 f2a -rwxrwxrwx 1 root root 0 Nov 18 19:05 f2b total 1 -rwxrwxrwx 1 root root 0 Nov 18 19:05 f2a -rwxrwxrwx 1 root root 0 Nov 18 19:05 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0110004 s, 46.5 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 19:05:26 (1763510726) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 19:05:28 (1763510728) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244237 s, 8.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 19:05:34 (1763510734) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 19:05:39 (1763510739) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 19:05:45 (1763510745) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 19:05:50 (1763510750) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 19:05:55 (1763510755) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 19:06:00 (1763510760) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 19:06:04 (1763510764) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 19:06:11 (1763510771) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576237 s, 18.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0484795 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 19:06:16 (1763510776) 1+0 records in 1+0 records out 1 byte copied, 0.00876315 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00883523 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109627 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00863981 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0131466 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0256545 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00960994 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119077 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101364 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123385 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0159717 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00933546 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 19:06:22 (1763510782) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0209623 s, 147 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0809851 s, 50.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000236871 s, 13.0 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 19:06:28 (1763510788) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 19:06:34 (1763510794) warning: '-runtime' deprecated, use '-t runtime' instead - link 400 (time 1763510806.54 total 10.05 last 39.80) - link 816 (time 1763510816.54 total 20.06 last 41.56) - link 1230 (time 1763510826.57 total 30.08 last 41.31) - link 1620 (time 1763510836.57 total 40.09 last 38.97) - link 2066 (time 1763510846.59 total 50.10 last 44.53) - link 2513 (time 1763510856.61 total 60.12 last 44.62) - link 2934 (time 1763510866.61 total 70.12 last 42.09) - link 3320 (time 1763510876.61 total 80.13 last 38.59) - link 3679 (time 1763510886.61 total 90.13 last 35.90) total: 4030 link in 100.01 seconds: 40.30 ops/second using seed 2920539217 running for 100 seconds - stat 8378 (time 1763510808 ; total 11 ; last 11) - stat 10000 (time 1763510809 ; total 12 ; last 1) - stat 19634 (time 1763510820 ; total 23 ; last 11) - stat 20000 (time 1763510820 ; total 23 ; last 0) - stat 28227 (time 1763510831 ; total 34 ; last 11) - stat 30000 (time 1763510833 ; total 36 ; last 2) - stat 39401 (time 1763510844 ; total 47 ; last 11) - stat 40000 (time 1763510844 ; total 47 ; last 0) - stat 45265 (time 1763510855 ; total 58 ; last 11) - stat 49548 (time 1763510866 ; total 69 ; last 11) - stat 50000 (time 1763510867 ; total 70 ; last 1) - stat 53636 (time 1763510878 ; total 81 ; last 11) - stat 58050 (time 1763510889 ; total 92 ; last 11) - stat 60000 (time 1763510893 ; total 96 ; last 4) total: 61302 stats in 100 seconds: 613.020020 stats/second - unlinked 0 (time 1763510847 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile4030) error: No such file or directory total: 4029 unlinks in 94 seconds: 42.861702 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21381) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21383) - No such process PASS 12 (151s) == sanityn test 13: test directory page revocation ======= 19:09:05 (1763510945) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 19:09:12 (1763510952) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 22569 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (5s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 19:09:17 (1763510957) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 23149 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (5s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 19:09:22 (1763510962) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23726 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 19:09:28 (1763510968) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24308 Terminated $DIR1/$tdir/sleep 60 PASS 14c (5s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 19:09:33 (1763510973) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24890 Terminated $DIR1/$tdir/sleep 60 PASS 14d (6s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 19:09:40 (1763510980) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7526400 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7526400KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (42s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 19:10:21 (1763511021) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.435202 s, 19.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.316076 s, 26.5 MB/s Chance of close/open is 1 in 50 Seed set to 1024 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000007[1] 1763511025.454801 write 0x029415 thru 0x02f93b (0x6527 bytes) 000009[0] 1763511025.490811 mapread 0x000e71 thru 0x008ec6 (0x8056 bytes) truncating to largest ever: 0x731fb2 000010[0] 1763511025.588928 trunc from 0x02f93c to 0x731fb1 (0x702676 bytes) 000011[1] 1763511025.674449 read 0x59f22c thru 0x5a7ef8 (0x8ccd bytes) 000013[1] 1763511025.747131 read 0x2b56ae thru 0x2bfbe7 (0xa53a bytes) 000015[1] 1763511025.773171 read 0x147646 thru 0x14d41c (0x5dd7 bytes) 000016[1] 1763511025.783701 read 0x05b6f6 thru 0x068449 (0xcd54 bytes) 000021[1] 1763511025.802559 mapwrite 0x2793ce thru 0x27c8f9 (0x352c bytes) truncating to largest ever: 0x7bf2be 000022[1] 1763511025.992583 trunc from 0x731fb2 to 0x7bf2bd (0x8d30c bytes) 000024[0] 1763511026.075621 mapread 0x4b4b6c thru 0x4bcd34 (0x81c9 bytes) 000030[0] 1763511026.240618 write 0x5242da thru 0x526513 (0x223a bytes) 000032[0] 1763511026.313303 write 0x53afc9 thru 0x53e9fa (0x3a32 bytes) 000034[0] 1763511026.357252 read 0x6c286f thru 0x6c5889 (0x301b bytes) 000035[1] 1763511026.385324 mapread 0x3a9bee thru 0x3b22a9 (0x86bc bytes) truncating to largest ever: 0x9780a5 000036[0] 1763511026.425359 trunc from 0x7bf2be to 0x9780a4 (0x1b8de7 bytes) 000037[0] 1763511026.475200 read 0x2ddc85 thru 0x2e571b (0x7a97 bytes) 000042[1] 1763511026.536927 write 0x37b435 thru 0x382ff0 (0x7bbc bytes) 000045[0] 1763511026.607415 mapread 0x5ce445 thru 0x5d1ed4 (0x3a90 bytes) 000047[0] 1763511026.725668 mapread 0x3908df thru 0x396388 (0x5aaa bytes) 000049[0] 1763511026.820720 write 0x365f06 thru 0x36f2b1 (0x93ac bytes) 000051[0] 1763511026.886102 mapread 0x324fa8 thru 0x3297c8 (0x4821 bytes) 000052[1] 1763511026.912204 write 0x6a3789 thru 0x6b28ed (0xf165 bytes) 000053[0] 1763511026.966536 trunc from 0x9780a5 to 0x1240b3 (0x853ff1 bytes) 000057[1] 1763511027.137107 trunc from 0x1240b4 to 0x87b087 (0x756fd4 bytes) 000058[1] 1763511027.257393 read 0x47d764 thru 0x480e89 (0x3726 bytes) 000059[1] 1763511027.274643 mapwrite 0x88c085 thru 0x88e2dc (0x2258 bytes) 000060[1] 1763511027.393526 mapwrite 0x91a11d thru 0x91eb94 (0x4a78 bytes) 000064[0] 1763511027.589096 read 0x00b901 thru 0x019a0a (0xe10a bytes) 000066[0] 1763511027.652144 trunc from 0x91eb95 to 0x5456c6 (0x3d94ce bytes) 000067[0] 1763511027.710061 mapwrite 0x0521e7 thru 0x0615c8 (0xf3e2 bytes) 000069[0] 1763511027.895532 write 0x5e2d77 thru 0x5e7e4a (0x50d4 bytes) 000071[1] 1763511027.933590 write 0x406b93 thru 0x40a6a0 (0x3b0e bytes) 000074[0] 1763511028.021889 trunc from 0x5e7e4b to 0x91ace1 (0x332e97 bytes) 000078[1] 1763511028.133800 write 0x2ef646 thru 0x2fa69c (0xb057 bytes) 000079[0] 1763511028.187986 trunc from 0x91ace2 to 0x5db1b3 (0x33fb2e bytes) 000083[0] 1763511028.316114 read 0x059df1 thru 0x064344 (0xa554 bytes) 000084[1] 1763511028.327957 mapwrite 0x33fb17 thru 0x34d722 (0xdc0c bytes) 000085[0] 1763511028.449190 read 0x402e49 thru 0x40fb72 (0xcd2a bytes) 000086[0] 1763511028.485370 read 0x0b0f40 thru 0x0b3eb9 (0x2f7a bytes) 000087[0] 1763511028.530969 read 0x46fc50 thru 0x478e0c (0x91bd bytes) 000088[0] 1763511028.540944 read 0x3074c6 thru 0x3103ec (0x8f27 bytes) 000089[1] 1763511028.548850 write 0x93497e thru 0x93523f (0x8c2 bytes) 000091[1] 1763511028.606020 mapread 0x027fa0 thru 0x02cf80 (0x4fe1 bytes) 000092[1] 1763511028.684719 mapwrite 0x33bf9e thru 0x347f3f (0xbfa2 bytes) 000093[0] 1763511028.810074 trunc from 0x935240 to 0x5fbb63 (0x3396dc bytes) 000095[0] 1763511028.881403 read 0x12eeab thru 0x139c5e (0xadb4 bytes) 000096[1] 1763511028.890440 trunc from 0x5fbb64 to 0x01485b (0x5e7308 bytes) 000098[0] 1763511028.951364 mapwrite 0x7e44e4 thru 0x7e5085 (0xba2 bytes) 000100[0] 1763511029.165258 write 0x908e13 thru 0x917153 (0xe341 bytes) 000101[0] 1763511029.176641 write 0x0412d3 thru 0x04b9a8 (0xa6d6 bytes) 000102[1] 1763511029.218389 read 0x3b9c6d thru 0x3bca36 (0x2dca bytes) 000105[0] 1763511029.270034 mapwrite 0x482e6e thru 0x48c33e (0x94d1 bytes) 000108[1] 1763511029.391846 write 0x5e3cc0 thru 0x5ecd2d (0x906e bytes) 000114[1] 1763511029.491621 write 0x584840 thru 0x5868f8 (0x20b9 bytes) 000116[1] 1763511029.520180 write 0x71efb3 thru 0x72827c (0x92ca bytes) 000117[1] 1763511029.546557 trunc from 0x917154 to 0x92460f (0xd4bc bytes) 000120[0] 1763511029.648205 trunc from 0x924610 to 0x51035e (0x4142b1 bytes) 000121[0] 1763511029.712013 mapwrite 0x3bfe7c thru 0x3c2401 (0x2586 bytes) 000122[0] 1763511029.819999 write 0x8c8154 thru 0x8d43c2 (0xc26f bytes) 000125[1] 1763511029.876423 mapwrite 0x32b35b thru 0x3311fb (0x5ea1 bytes) 000126[1] 1763511030.012650 mapwrite 0x6d8730 thru 0x6dbe95 (0x3766 bytes) 000127[0] 1763511030.129047 mapwrite 0x557018 thru 0x561942 (0xa92b bytes) 000128[1] 1763511030.292148 trunc from 0x8d43c3 to 0x18951f (0x74aea3 bytes) 000132[0] 1763511030.380985 read 0x056d9a thru 0x05fc70 (0x8ed7 bytes) 000134[1] 1763511030.495193 mapread 0x1640e2 thru 0x1678de (0x37fd bytes) 000135[0] 1763511030.525655 mapread 0x177c7d thru 0x17cf8b (0x530f bytes) 000136[1] 1763511030.550306 read 0x0ab0dc thru 0x0b4457 (0x937c bytes) 000137[1] 1763511030.611697 write 0x16192f thru 0x16ff8c (0xe65e bytes) 000139[0] 1763511030.690489 mapwrite 0x8c9147 thru 0x8cca37 (0x38f1 bytes) 000144[1] 1763511031.046456 trunc from 0x8cca38 to 0x94b903 (0x7eecc bytes) 000145[0] 1763511031.112662 mapread 0x8be561 thru 0x8caae7 (0xc587 bytes) 000146[1] 1763511031.225208 read 0x242bc4 thru 0x246106 (0x3543 bytes) 000147[1] 1763511031.265444 write 0x317043 thru 0x323997 (0xc955 bytes) 000148[0] 1763511031.336294 write 0x3babb9 thru 0x3ca550 (0xf998 bytes) 000152[1] 1763511031.487519 write 0x63b89e thru 0x64135c (0x5abf bytes) 000153[0] 1763511031.535860 read 0x1a8590 thru 0x1aad60 (0x27d1 bytes) 000155[1] 1763511031.574173 trunc from 0x94b904 to 0x1f5d94 (0x755b6f bytes) 000157[1] 1763511031.666499 mapwrite 0x7ad1cc thru 0x7aee22 (0x1c57 bytes) 000159[1] 1763511031.833594 mapread 0x05d8d4 thru 0x06b44e (0xdb7b bytes) 000164[1] 1763511031.995060 mapwrite 0x47b53c thru 0x48187b (0x6340 bytes) 000165[0] 1763511032.070713 trunc from 0x7aee23 to 0x70359c (0xab886 bytes) 000166[0] 1763511032.126808 mapread 0x334be9 thru 0x3382e2 (0x36fa bytes) 000168[0] 1763511032.218434 trunc from 0x70359d to 0x02aec0 (0x6d86dc bytes) 000169[1] 1763511032.268271 write 0x3681e6 thru 0x375c04 (0xda1f bytes) 000171[0] 1763511032.314832 mapread 0x0e0c37 thru 0x0e2e33 (0x21fd bytes) 000173[1] 1763511032.402295 mapread 0x35b865 thru 0x35f945 (0x40e1 bytes) 000175[1] 1763511032.449615 mapread 0x1292ac thru 0x12b7e9 (0x253e bytes) 000180[1] 1763511032.506374 trunc from 0x375c05 to 0x2495cd (0x12c637 bytes) 000181[1] 1763511032.570053 mapread 0x05ccd9 thru 0x05f889 (0x2bb1 bytes) 000182[1] 1763511032.616611 mapwrite 0x327228 thru 0x32ff36 (0x8d0f bytes) 000183[0] 1763511032.764679 read 0x14cea8 thru 0x14f657 (0x27b0 bytes) 000184[0] 1763511032.858816 mapwrite 0x6d3488 thru 0x6de696 (0xb20f bytes) 000186[1] 1763511033.088656 mapwrite 0x59a95e thru 0x5a2c62 (0x8305 bytes) 000190[1] 1763511033.299844 read 0x365f63 thru 0x36d291 (0x732f bytes) 000191[0] 1763511033.344600 trunc from 0x6de697 to 0x8c45cb (0x1e5f35 bytes) 000193[1] 1763511033.409237 mapread 0x27c59a thru 0x27e6ec (0x2153 bytes) 000198[0] 1763511033.512676 write 0x1079d6 thru 0x1104b7 (0x8ae2 bytes) 000200[0] 1763511033.549122 read 0x4f3c8b thru 0x50224f (0xe5c5 bytes) 000201[0] 1763511033.591440 trunc from 0x8c45cc to 0x101f09 (0x7c26c2 bytes) 000202[1] 1763511033.622091 mapwrite 0x98ae40 thru 0x997870 (0xca31 bytes) 000205[1] 1763511033.828435 read 0x45ffa3 thru 0x4661e8 (0x6246 bytes) 000206[1] 1763511033.841646 write 0x7a6952 thru 0x7a7d5e (0x140d bytes) 000208[0] 1763511033.894440 mapwrite 0x4abf40 thru 0x4b88c8 (0xc989 bytes) 000209[1] 1763511034.082297 mapread 0x159086 thru 0x162428 (0x93a3 bytes) 000210[1] 1763511034.163511 trunc from 0x997871 to 0x8c9e88 (0xcd9e8 bytes) 000211[1] 1763511034.244078 mapwrite 0x00290e thru 0x0084dd (0x5bd0 bytes) 000214[0] 1763511034.381988 trunc from 0x8c9e89 to 0x275aa4 (0x6543e4 bytes) 000215[0] 1763511034.421790 read 0x0c63e6 thru 0x0d1e2f (0xba4a bytes) 000219[0] 1763511034.515016 mapwrite 0x253917 thru 0x25ac39 (0x7323 bytes) 000220[1] 1763511034.602304 trunc from 0x275aa5 to 0x24c9f2 (0x290b2 bytes) 000221[0] 1763511034.655219 read 0x1afe8c thru 0x1bc569 (0xc6de bytes) 000225[1] 1763511034.829677 trunc from 0x24c9f3 to 0x5cb51e (0x37eb2c bytes) 000227[0] 1763511034.882584 mapwrite 0x79ddba thru 0x7a5c1a (0x7e61 bytes) 000229[0] 1763511035.090892 mapwrite 0x90a567 thru 0x90d757 (0x31f1 bytes) 000230[0] 1763511035.210589 read 0x3f75ac thru 0x40447a (0xcecf bytes) 000233[0] 1763511035.237795 trunc from 0x90d758 to 0x560e8f (0x3ac8c8 bytes) 000234[0] 1763511035.265846 mapwrite 0x6ed60c thru 0x6fcee4 (0xf8d9 bytes) 000237[1] 1763511035.407196 trunc from 0x6fcee5 to 0x31b667 (0x3e187d bytes) 000238[0] 1763511035.464209 trunc from 0x31b668 to 0x6a69f0 (0x38b389 bytes) 000240[1] 1763511035.527067 trunc from 0x6a69f1 to 0x79c336 (0xf5946 bytes) 000242[0] 1763511035.582148 write 0x0888d8 thru 0x092db8 (0xa4e1 bytes) 000244[1] 1763511035.646460 mapread 0x0f0ff5 thru 0x0f6d31 (0x5d3d bytes) 000246[1] 1763511035.736823 read 0x33223a thru 0x3353ae (0x3175 bytes) 000247[1] 1763511035.744918 write 0x9be14d thru 0x9c479b (0x664f bytes) 000248[1] 1763511035.766985 mapread 0x25df05 thru 0x2650fd (0x71f9 bytes) 000249[1] 1763511035.851465 mapread 0x8e0257 thru 0x8ef9ed (0xf797 bytes) 000251[0] 1763511035.966491 mapread 0x0af779 thru 0x0b9ebe (0xa746 bytes) 000253[0] 1763511036.044788 mapread 0x7469c0 thru 0x74a7c7 (0x3e08 bytes) 000255[1] 1763511036.107599 read 0x0c6fac thru 0x0c7b7a (0xbcf bytes) 000256[1] 1763511036.113194 read 0x9a0346 thru 0x9ac79c (0xc457 bytes) 000257[1] 1763511036.120140 write 0x66bf0e thru 0x673c14 (0x7d07 bytes) 000259[1] 1763511036.192461 mapread 0x66320c thru 0x665cef (0x2ae4 bytes) 000262[1] 1763511036.230757 mapwrite 0x1a44da thru 0x1a57ff (0x1326 bytes) 000263[0] 1763511036.319820 mapwrite 0x19d695 thru 0x1a2053 (0x49bf bytes) 000264[1] 1763511036.432824 write 0x5982dc thru 0x59e963 (0x6688 bytes) 000265[1] 1763511036.459043 trunc from 0x9c479c to 0x5829d0 (0x441dcb bytes) 000267[1] 1763511036.524509 trunc from 0x5829d1 to 0x347c22 (0x23adae bytes) 000269[0] 1763511036.572751 trunc from 0x347c23 to 0x8b9dfb (0x5721d9 bytes) 000270[0] 1763511036.623312 read 0x535cdc thru 0x5418c8 (0xbbed bytes) 000272[0] 1763511036.640188 trunc from 0x8b9dfc to 0x946b8b (0x8cd90 bytes) 000275[1] 1763511036.716358 write 0x451bc9 thru 0x458919 (0x6d51 bytes) 000278[1] 1763511036.806543 mapwrite 0x0165cd thru 0x01b0f6 (0x4b2a bytes) 000279[0] 1763511036.892248 trunc from 0x946b8c to 0x66bff5 (0x2dab96 bytes) 000282[1] 1763511037.008471 write 0x9f158b thru 0x9ff798 (0xe20e bytes) 000283[0] 1763511037.048681 mapread 0x808082 thru 0x80efc7 (0x6f46 bytes) 000286[1] 1763511037.111400 mapwrite 0x8a5341 thru 0x8a9def (0x4aaf bytes) 000287[0] 1763511037.249337 read 0x6ab582 thru 0x6b43b3 (0x8e32 bytes) 000288[0] 1763511037.342573 mapread 0x839b0f thru 0x83c5b4 (0x2aa6 bytes) 000290[0] 1763511037.372628 trunc from 0x9ff799 to 0x446631 (0x5b9167 bytes) 000291[0] 1763511037.410278 trunc from 0x446632 to 0x35d734 (0xe8efd bytes) 000292[1] 1763511037.453025 trunc from 0x35d735 to 0x17336e (0x1ea3c6 bytes) 000297[0] 1763511037.582677 mapread 0x0e3c20 thru 0x0f1b52 (0xdf33 bytes) 000298[0] 1763511037.639303 mapwrite 0x39d412 thru 0x39fa10 (0x25ff bytes) truncating to largest ever: 0x9cf0e9 000300[1] 1763511037.771038 trunc from 0x39fa11 to 0x9cf0e8 (0x62f6d8 bytes) 000307[1] 1763511037.896680 trunc from 0x9cf0e9 to 0x302902 (0x6cc7e6 bytes) 000311[0] 1763511037.988090 trunc from 0x302903 to 0x9bb435 (0x6b8b33 bytes) 000313[1] 1763511038.067080 mapwrite 0x3b446d thru 0x3c33ca (0xef5e bytes) 000314[1] 1763511038.200474 write 0x375e2b thru 0x379419 (0x35ef bytes) 000315[1] 1763511038.226740 mapwrite 0x3721ac thru 0x380f8b (0xede0 bytes) 000320[0] 1763511038.413810 trunc from 0x9bb436 to 0x444b4a (0x5768eb bytes) 000321[1] 1763511038.456655 read 0x237f5b thru 0x243873 (0xb919 bytes) 000322[1] 1763511038.503196 mapread 0x329045 thru 0x32f117 (0x60d3 bytes) 000326[0] 1763511038.620921 trunc from 0x444b4b to 0x357f42 (0xecc08 bytes) 000327[1] 1763511038.666145 read 0x066b84 thru 0x06dee0 (0x735d bytes) 000331[1] 1763511038.736475 read 0x16d96f thru 0x173a7a (0x610c bytes) 000335[0] 1763511038.833833 trunc from 0x357f43 to 0x4eb19c (0x19325a bytes) 000336[0] 1763511038.877462 trunc from 0x4eb19d to 0x7856b5 (0x29a519 bytes) 000337[1] 1763511038.911561 read 0x24c443 thru 0x256e4f (0xaa0d bytes) 000338[1] 1763511038.941118 write 0x296479 thru 0x29f2dd (0x8e65 bytes) 000340[0] 1763511039.011696 mapwrite 0x6d71e8 thru 0x6db7e0 (0x45f9 bytes) 000341[1] 1763511039.127344 read 0x0c21ba thru 0x0c2bbd (0xa04 bytes) 000342[0] 1763511039.172782 trunc from 0x7856b6 to 0x0f8bcc (0x68cae9 bytes) 000343[0] 1763511039.228657 write 0x3f0c0c thru 0x3fcb51 (0xbf46 bytes) 000346[1] 1763511039.260381 write 0x6b2b4e thru 0x6bfcee (0xd1a1 bytes) 000347[0] 1763511039.285815 trunc from 0x6bfcef to 0x144655 (0x57b699 bytes) 000349[1] 1763511039.333682 write 0x8e6971 thru 0x8f1151 (0xa7e1 bytes) 000352[0] 1763511039.392185 mapwrite 0x23d5e5 thru 0x2463e0 (0x8dfc bytes) 000355[0] 1763511039.511155 mapread 0x38ca3f thru 0x399bcc (0xd18e bytes) 000356[1] 1763511039.543228 trunc from 0x8f1152 to 0x24d2ad (0x6a3ea4 bytes) 000357[0] 1763511039.589523 mapwrite 0x7d5f5d thru 0x7e3c16 (0xdcba bytes) 000359[0] 1763511039.747648 read 0x33516e thru 0x342066 (0xcef9 bytes) 000360[1] 1763511039.781991 trunc from 0x7e3c17 to 0x62ca70 (0x1b71a6 bytes) 000364[1] 1763511039.820379 write 0x1130ce thru 0x1203dc (0xd30f bytes) 000365[0] 1763511039.846323 read 0x3c9640 thru 0x3cf079 (0x5a3a bytes) 000366[0] 1763511039.889186 trunc from 0x62ca71 to 0x39f094 (0x28d9dc bytes) 000371[0] 1763511039.994276 mapwrite 0x2b344e thru 0x2bfff5 (0xcba8 bytes) 000373[0] 1763511040.126245 write 0x944101 thru 0x95406a (0xff6a bytes) 000374[0] 1763511040.158131 mapwrite 0x88c2af thru 0x88fba4 (0x38f6 bytes) 000379[1] 1763511040.254220 trunc from 0x95406b to 0x35c9ee (0x5f767c bytes) 000380[0] 1763511040.310831 write 0x6c4298 thru 0x6c972a (0x5493 bytes) 000381[0] 1763511040.351985 mapread 0x0419cd thru 0x04ee8d (0xd4c1 bytes) 000382[1] 1763511040.410212 mapwrite 0x8b7f70 thru 0x8beae0 (0x6b71 bytes) 000384[1] 1763511040.593365 write 0x635b80 thru 0x63e4e7 (0x8968 bytes) 000385[1] 1763511040.605063 mapread 0x6c6475 thru 0x6d39f6 (0xd582 bytes) 000391[1] 1763511040.686705 mapwrite 0x84cc9f thru 0x8539f6 (0x6d58 bytes) 000394[0] 1763511040.813402 read 0x07ebce thru 0x08a246 (0xb679 bytes) 000396[1] 1763511040.853121 write 0x22cda3 thru 0x22dda8 (0x1006 bytes) 000397[0] 1763511040.875206 mapwrite 0x777f28 thru 0x77cd25 (0x4dfe bytes) 000398[0] 1763511041.009655 write 0x11538f thru 0x1200c9 (0xad3b bytes) 000399[0] 1763511041.058518 write 0x825387 thru 0x82ad6b (0x59e5 bytes) 000402[0] 1763511041.107273 read 0x768dd2 thru 0x778470 (0xf69f bytes) 000404[1] 1763511041.115451 read 0x010dec thru 0x0197da (0x89ef bytes) 000406[0] 1763511041.169392 mapwrite 0x31f93c thru 0x320786 (0xe4b bytes) 000407[0] 1763511041.260478 read 0x547a82 thru 0x548b28 (0x10a7 bytes) 000409[1] 1763511041.311037 read 0x1fb833 thru 0x2085bf (0xcd8d bytes) 000410[1] 1763511041.342236 trunc from 0x8beae1 to 0x0623f6 (0x85c6ea bytes) 000411[0] 1763511041.423870 mapread 0x009a48 thru 0x0126c2 (0x8c7b bytes) 000415[1] 1763511041.515730 mapread 0x04e687 thru 0x05af80 (0xc8fa bytes) 000416[1] 1763511041.565972 read 0x01388a thru 0x01675b (0x2ed2 bytes) 000417[0] 1763511041.578683 trunc from 0x0623f7 to 0x53aee1 (0x4d8aeb bytes) 000418[0] 1763511041.673005 trunc from 0x53aee2 to 0x12bae4 (0x40f3fd bytes) 000419[1] 1763511041.705630 write 0x118222 thru 0x11a167 (0x1f46 bytes) 000424[0] 1763511041.852851 mapread 0x10718b thru 0x108134 (0xfaa bytes) 000425[0] 1763511041.899067 trunc from 0x12bae5 to 0x3c515d (0x299679 bytes) 000428[1] 1763511041.949982 mapwrite 0x80095d thru 0x810915 (0xffb9 bytes) 000430[1] 1763511042.172155 read 0x306fd1 thru 0x315d87 (0xedb7 bytes) 000431[1] 1763511042.172817 write 0x02a490 thru 0x02c64b (0x21bc bytes) 000433[0] 1763511042.186168 trunc from 0x810916 to 0x0e0b9c (0x72fd79 bytes) 000434[1] 1763511042.269600 read 0x0ce837 thru 0x0d98d5 (0xb09f bytes) 000437[0] 1763511042.381068 trunc from 0x0e0b9d to 0x864d19 (0x78417d bytes) 000439[1] 1763511042.423150 mapread 0x52d93a thru 0x52f078 (0x173f bytes) 000440[1] 1763511042.474544 mapwrite 0x8bc679 thru 0x8bf9d2 (0x335a bytes) 000441[0] 1763511042.659723 mapread 0x769d27 thru 0x776960 (0xcc3a bytes) 000443[0] 1763511042.719185 read 0x47b84e thru 0x48783a (0xbfed bytes) 000447[1] 1763511042.786525 trunc from 0x8bf9d3 to 0x1da3c0 (0x6e5612 bytes) 000448[0] 1763511042.835798 read 0x0a32fc thru 0x0a66f7 (0x33fc bytes) 000451[0] 1763511042.901188 read 0x011bb5 thru 0x019b88 (0x7fd4 bytes) 000453[0] 1763511042.919345 write 0x44bc20 thru 0x45316f (0x7550 bytes) 000454[0] 1763511042.948530 mapread 0x39dec1 thru 0x3a5647 (0x7787 bytes) 000455[1] 1763511042.999443 trunc from 0x453170 to 0x5d4158 (0x180fe9 bytes) 000462[0] 1763511043.130405 read 0x0e8a15 thru 0x0f7bc3 (0xf1af bytes) 000463[0] 1763511043.163930 mapwrite 0x258dce thru 0x25ad03 (0x1f36 bytes) 000464[0] 1763511043.254105 mapwrite 0x4ddb8a thru 0x4e2d9d (0x5214 bytes) 000467[0] 1763511043.377575 read 0x3d64aa thru 0x3dba24 (0x557b bytes) 000468[1] 1763511043.397510 mapread 0x335930 thru 0x33c138 (0x6809 bytes) 000471[0] 1763511043.503439 write 0x0537d4 thru 0x05e6c2 (0xaeef bytes) 000478[0] 1763511043.585826 mapwrite 0x1ad049 thru 0x1bbb06 (0xeabe bytes) 000481[0] 1763511043.693046 trunc from 0x5d4159 to 0x40d37c (0x1c6ddc bytes) 000482[0] 1763511043.724933 write 0x24884f thru 0x248949 (0xfb bytes) 000483[1] 1763511043.751381 trunc from 0x40d37d to 0x3f67ee (0x16b8e bytes) 000485[1] 1763511043.823791 mapwrite 0x340402 thru 0x348ffe (0x8bfd bytes) 000486[0] 1763511043.920636 mapwrite 0x452933 thru 0x45d1ad (0xa87b bytes) 000487[1] 1763511044.056234 mapwrite 0x54553d thru 0x5497c9 (0x428d bytes) 000488[0] 1763511044.194833 mapread 0x1c2a4f thru 0x1c301f (0x5d1 bytes) 000489[0] 1763511044.228490 trunc from 0x5497ca to 0x9981df (0x44ea16 bytes) 000490[0] 1763511044.302234 write 0x7b08f8 thru 0x7b66b7 (0x5dc0 bytes) 000492[0] 1763511044.363143 trunc from 0x9981e0 to 0x31b04f (0x67d190 bytes) 000493[1] 1763511044.387259 mapread 0x2c567e thru 0x2d3073 (0xd9f6 bytes) 000496[0] 1763511044.473367 read 0x1fe831 thru 0x20a6df (0xbeaf bytes) 000497[1] 1763511044.557163 mapwrite 0x95cca1 thru 0x966395 (0x96f5 bytes) 000499[0] 1763511044.661304 write 0x1d5904 thru 0x1e1645 (0xbd42 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1044 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[0] 1763511045.401698 mapwrite 0x3de000 thru 0x3eab4e (0xcb4f bytes) 000005[0] 1763511045.531025 mapwrite 0x7a0000 thru 0x7a0f7f (0xf80 bytes) 000007[1] 1763511045.623837 mapwrite 0x4ae000 thru 0x4b0025 (0x2026 bytes) 000009[0] 1763511045.708740 mapwrite 0x929000 thru 0x930dce (0x7dcf bytes) truncating to largest ever: 0x2dd8c9 000011[0] 1763511045.839890 trunc from 0x930dcf to 0x2dd8c8 (0x653506 bytes) 000013[1] 1763511045.882417 mapread 0x1a7000 thru 0x1b2319 (0xb31a bytes) truncating to largest ever: 0x670d93 000014[1] 1763511045.929529 trunc from 0x2dd8c9 to 0x670d92 (0x3934ca bytes) 000016[1] 1763511045.994182 mapwrite 0x509000 thru 0x514552 (0xb553 bytes) 000017[0] 1763511046.105439 mapread 0x64a000 thru 0x6568cf (0xc8d0 bytes) 000018[1] 1763511046.166088 write 0x86b000 thru 0x87701b (0xc01c bytes) 000022[0] 1763511046.192721 mapwrite 0x9b1000 thru 0x9b6e17 (0x5e18 bytes) 000023[0] 1763511046.317986 read_OD 0x0dc000 thru 0x0e4fff (0x9000 bytes) 000025[1] 1763511046.345149 read 0x351000 thru 0x352837 (0x1838 bytes) 000026[1] 1763511046.365196 read 0x663000 thru 0x66a881 (0x7882 bytes) 000027[1] 1763511046.377168 read 0x7df000 thru 0x7e6c3a (0x7c3b bytes) 000028[0] 1763511046.388265 mapwrite 0x433000 thru 0x43c9b3 (0x99b4 bytes) 000029[0] 1763511046.471066 mapread 0x87c000 thru 0x888636 (0xc637 bytes) 000030[1] 1763511046.511099 trunc from 0x9b6e18 to 0x40405b (0x5b2dbc bytes) 000031[0] 1763511046.555025 read_OD 0x0fb000 thru 0x0fcfff (0x2000 bytes) truncating to largest ever: 0x8ee80b 000032[0] 1763511046.584542 trunc from 0x40405c to 0x8ee80a (0x4ea7af bytes) 000036[0] 1763511046.668494 mapread 0x215000 thru 0x21b80c (0x680d bytes) 000037[1] 1763511046.686239 write 0x987000 thru 0x992a5a (0xba5b bytes) 000038[0] 1763511046.695003 mapwrite 0x91a000 thru 0x923973 (0x9974 bytes) 000039[1] 1763511046.821228 mapwrite 0x576000 thru 0x57aacf (0x4ad0 bytes) 000040[0] 1763511046.940402 write_OD 0x8bf000 thru 0x8c3fff (0x5000 bytes) 000043[1] 1763511047.000270 write 0x02a000 thru 0x02cf9a (0x2f9b bytes) 000045[1] 1763511047.019521 trunc from 0x992a5b to 0x8acfbd (0xe5a9d bytes) 000047[1] 1763511047.061811 read 0x208000 thru 0x20afad (0x2fae bytes) 000049[1] 1763511047.067258 mapread 0x0a4000 thru 0x0ab566 (0x7567 bytes) 000052[0] 1763511047.116497 trunc from 0x8acfbe to 0x6af1e8 (0x1fddd5 bytes) 000056[0] 1763511047.195459 mapread 0x0f7000 thru 0x0ffd43 (0x8d44 bytes) truncating to largest ever: 0x8f8ada 000057[1] 1763511047.228592 trunc from 0x6af1e9 to 0x8f8ad9 (0x2498f1 bytes) 000059[1] 1763511047.302892 read 0x4c0000 thru 0x4ce331 (0xe332 bytes) 000060[1] 1763511047.321084 mapread 0x33a000 thru 0x33f5e2 (0x55e3 bytes) 000061[0] 1763511047.362181 mapwrite 0x456000 thru 0x463e97 (0xde98 bytes) 000063[1] 1763511047.493423 trunc from 0x8f8ada to 0x2c605c (0x632a7d bytes) 000064[0] 1763511047.536439 write_OD 0x024000 thru 0x02dfff (0xa000 bytes) 000065[0] 1763511047.587972 mapwrite 0x59f000 thru 0x5a5250 (0x6251 bytes) 000066[1] 1763511047.739546 mapread 0x2c6000 thru 0x2ca777 (0x4778 bytes) 000067[0] 1763511047.784296 trunc from 0x5a5251 to 0x5ff43d (0x5a1ed bytes) 000068[0] 1763511047.825588 read_OD 0x06b000 thru 0x071fff (0x7000 bytes) 000069[0] 1763511047.845277 write_OD 0x7d8000 thru 0x7d9fff (0x2000 bytes) 000070[1] 1763511047.892376 trunc from 0x7da000 to 0x32dfdf (0x4ac020 bytes) 000072[0] 1763511047.923361 trunc from 0x32dfe0 to 0x105ad6 (0x228509 bytes) 000074[0] 1763511047.953754 read_OD 0x058000 thru 0x05ffff (0x8000 bytes) 000077[1] 1763511047.981802 mapwrite 0x1b3000 thru 0x1bbffe (0x8fff bytes) 000078[0] 1763511048.158415 trunc from 0x1bbfff to 0x89a14f (0x6de151 bytes) 000080[0] 1763511048.232138 write_OD 0x83f000 thru 0x845fff (0x7000 bytes) 000081[1] 1763511048.287103 mapwrite 0x05a000 thru 0x0654f8 (0xb4f9 bytes) 000084[0] 1763511048.467397 trunc from 0x89a150 to 0x0ec1b2 (0x7adf9d bytes) 000086[0] 1763511048.509052 mapread 0x04a000 thru 0x04c0e9 (0x20ea bytes) 000089[0] 1763511048.577700 trunc from 0x0ec1b3 to 0x04f413 (0x9cd9f bytes) 000090[1] 1763511048.625681 mapread 0x04a000 thru 0x04e230 (0x4231 bytes) 000092[0] 1763511048.659506 mapwrite 0x4d8000 thru 0x4e47fb (0xc7fc bytes) 000095[1] 1763511048.902512 read 0x181000 thru 0x18d7d9 (0xc7da bytes) 000100[1] 1763511048.953212 read 0x039000 thru 0x03b367 (0x2368 bytes) 000101[0] 1763511048.965292 write_OD 0x267000 thru 0x26bfff (0x5000 bytes) 000102[0] 1763511049.052323 trunc from 0x4e47fc to 0x15ede3 (0x385a18 bytes) 000105[0] 1763511049.127463 mapwrite 0x209000 thru 0x2123bd (0x93be bytes) 000108[0] 1763511049.292024 mapread 0x1e1000 thru 0x1eff5f (0xef60 bytes) 000109[1] 1763511049.402730 mapwrite 0x641000 thru 0x641ef5 (0xef6 bytes) 000110[1] 1763511049.488982 mapread 0x184000 thru 0x189e66 (0x5e67 bytes) 000112[0] 1763511049.530694 read_OD 0x2cb000 thru 0x2d8fff (0xe000 bytes) 000113[0] 1763511049.579253 write_OD 0x418000 thru 0x41cfff (0x5000 bytes) 000114[0] 1763511049.626002 read_OD 0x52f000 thru 0x530fff (0x2000 bytes) truncating to largest ever: 0x8fbac2 000115[0] 1763511049.638631 trunc from 0x641ef6 to 0x8fbac1 (0x2b9bcc bytes) 000116[0] 1763511049.683868 mapread 0x151000 thru 0x151c08 (0xc09 bytes) 000117[0] 1763511049.714961 read_OD 0x01b000 thru 0x023fff (0x9000 bytes) 000122[0] 1763511049.750502 mapread 0x1db000 thru 0x1ddc9e (0x2c9f bytes) 000128[1] 1763511049.779536 mapwrite 0x4a0000 thru 0x4a17a2 (0x17a3 bytes) 000130[1] 1763511049.841733 read 0x7a5000 thru 0x7b40de (0xf0df bytes) 000134[1] 1763511049.870681 mapwrite 0x320000 thru 0x326a98 (0x6a99 bytes) 000135[1] 1763511049.961711 write 0x6a5000 thru 0x6abaa7 (0x6aa8 bytes) 000138[0] 1763511050.025294 read_OD 0x664000 thru 0x668fff (0x5000 bytes) 000139[1] 1763511050.064146 trunc from 0x8fbac2 to 0x365444 (0x59667d bytes) 000143[0] 1763511050.163355 mapwrite 0x33a000 thru 0x33e209 (0x420a bytes) 000144[0] 1763511050.270317 mapread 0x1e2000 thru 0x1e6bad (0x4bae bytes) 000145[0] 1763511050.305858 mapread 0x194000 thru 0x19a648 (0x6649 bytes) 000147[1] 1763511050.365365 trunc from 0x365445 to 0x24ba20 (0x119a24 bytes) 000149[0] 1763511050.436483 mapread 0x060000 thru 0x06eabe (0xeabf bytes) 000151[1] 1763511050.506402 mapread 0x0fe000 thru 0x10398e (0x598f bytes) 000154[1] 1763511050.536329 mapwrite 0x841000 thru 0x850cd2 (0xfcd3 bytes) 000155[0] 1763511050.656429 write_OD 0x7b0000 thru 0x7b5fff (0x6000 bytes) 000156[1] 1763511050.708348 read 0x469000 thru 0x46d4ae (0x44af bytes) 000157[0] 1763511050.720493 write_OD 0x9b7000 thru 0x9bafff (0x4000 bytes) 000159[0] 1763511050.778593 trunc from 0x9bb000 to 0x2292ff (0x791d00 bytes) 000162[0] 1763511050.852593 mapwrite 0x986000 thru 0x994f29 (0xef2a bytes) 000163[1] 1763511050.947615 mapread 0x295000 thru 0x29f602 (0xa603 bytes) 000165[0] 1763511051.002124 read_OD 0x57f000 thru 0x58dfff (0xf000 bytes) 000166[0] 1763511051.010870 read_OD 0x014000 thru 0x014fff (0x1000 bytes) 000168[1] 1763511051.016188 trunc from 0x994f2a to 0x625394 (0x36fb95 bytes) truncating to largest ever: 0x9ec294 000169[1] 1763511051.063748 trunc from 0x625395 to 0x9ec293 (0x3c6eff bytes) 000171[0] 1763511051.136746 trunc from 0x9ec294 to 0x944816 (0xa7a7d bytes) 000173[1] 1763511051.159103 mapwrite 0x8eb000 thru 0x8f9ad9 (0xeada bytes) 000174[1] 1763511051.269113 mapwrite 0x28e000 thru 0x2997f0 (0xb7f1 bytes) 000176[0] 1763511051.425436 trunc from 0x944817 to 0x18a8a8 (0x7b9f6e bytes) 000180[0] 1763511051.508152 mapwrite 0x6b3000 thru 0x6b8fc3 (0x5fc4 bytes) 000181[0] 1763511051.617951 trunc from 0x6b8fc4 to 0x16a86a (0x54e759 bytes) 000182[0] 1763511051.655713 write_OD 0x1b3000 thru 0x1b8fff (0x6000 bytes) 000185[0] 1763511051.722430 trunc from 0x1b9000 to 0x28df7f (0xd4f80 bytes) 000186[0] 1763511051.755463 write_OD 0x934000 thru 0x93dfff (0xa000 bytes) 000188[1] 1763511051.806516 trunc from 0x93e000 to 0x5587c1 (0x3e583e bytes) 000194[0] 1763511051.878113 mapwrite 0x6ab000 thru 0x6b68e4 (0xb8e5 bytes) 000196[1] 1763511052.036112 mapread 0x0ce000 thru 0x0dc8f8 (0xe8f9 bytes) 000197[1] 1763511052.116673 mapread 0x5c9000 thru 0x5ced17 (0x5d18 bytes) 000199[1] 1763511052.145685 read 0x08e000 thru 0x09114a (0x314b bytes) 000202[0] 1763511052.152192 trunc from 0x6b68e5 to 0x3d1547 (0x2e539d bytes) 000204[1] 1763511052.240271 read 0x01a000 thru 0x0255fa (0xb5fb bytes) 000205[1] 1763511052.315767 write 0x19e000 thru 0x1a9b8e (0xbb8f bytes) 000206[1] 1763511052.369069 trunc from 0x3d1548 to 0x02cced (0x3a485a bytes) 000207[0] 1763511052.392731 read_OD 0x020000 thru 0x026fff (0x7000 bytes) 000209[1] 1763511052.442755 mapwrite 0x445000 thru 0x44fd06 (0xad07 bytes) 000210[1] 1763511052.560094 mapread 0x1da000 thru 0x1e98ca (0xf8cb bytes) 000211[0] 1763511052.697791 mapread 0x3ac000 thru 0x3bb3d2 (0xf3d3 bytes) 000212[0] 1763511052.803667 mapwrite 0x091000 thru 0x096ec8 (0x5ec9 bytes) 000214[0] 1763511052.986529 mapread 0x242000 thru 0x24ecd5 (0xccd6 bytes) 000216[1] 1763511053.053598 trunc from 0x44fd07 to 0x1f1e0b (0x25defb bytes) 000217[1] 1763511053.117062 mapwrite 0x948000 thru 0x957c98 (0xfc99 bytes) 000220[0] 1763511053.366955 mapread 0x4ae000 thru 0x4b9eeb (0xbeec bytes) 000221[0] 1763511053.490389 mapread 0x7d3000 thru 0x7d970f (0x6710 bytes) 000222[1] 1763511053.524554 write 0x689000 thru 0x6939ac (0xa9ad bytes) 000223[1] 1763511053.553498 trunc from 0x957c99 to 0x398d71 (0x5bef27 bytes) 000225[0] 1763511053.602207 mapwrite 0x8e6000 thru 0x8ea3cd (0x43ce bytes) 000229[1] 1763511053.828428 mapread 0x228000 thru 0x23633a (0xe33b bytes) 000236[0] 1763511054.004238 mapread 0x3f0000 thru 0x3fb5c1 (0xb5c2 bytes) 000237[1] 1763511054.076536 read 0x157000 thru 0x15c8ea (0x58eb bytes) 000238[0] 1763511054.085327 mapread 0x7df000 thru 0x7e477f (0x5780 bytes) 000239[1] 1763511054.136180 write 0x141000 thru 0x1502ae (0xf2af bytes) 000240[0] 1763511054.219084 trunc from 0x8ea3ce to 0x8f5bb3 (0xb7e6 bytes) 000241[1] 1763511054.276255 read 0x62a000 thru 0x634054 (0xa055 bytes) 000243[1] 1763511054.307415 trunc from 0x8f5bb4 to 0x94045d (0x4a8aa bytes) 000245[0] 1763511054.373592 mapwrite 0x155000 thru 0x160886 (0xb887 bytes) 000246[0] 1763511054.540416 read_OD 0x3f4000 thru 0x3fdfff (0xa000 bytes) 000248[1] 1763511054.559372 mapread 0x595000 thru 0x599d41 (0x4d42 bytes) 000249[1] 1763511054.582511 write 0x40f000 thru 0x41daad (0xeaae bytes) 000250[0] 1763511054.616453 write_OD 0x6d6000 thru 0x6dafff (0x5000 bytes) 000253[1] 1763511054.674813 write 0x2ff000 thru 0x304555 (0x5556 bytes) 000255[0] 1763511054.700366 mapwrite 0x209000 thru 0x215f68 (0xcf69 bytes) 000256[0] 1763511054.834330 read_OD 0x501000 thru 0x509fff (0x9000 bytes) 000257[0] 1763511054.863621 trunc from 0x94045e to 0x38518f (0x5bb2ce bytes) 000258[1] 1763511054.948206 mapread 0x219000 thru 0x219956 (0x957 bytes) 000259[0] 1763511055.003145 mapread 0x00a000 thru 0x019c21 (0xfc22 bytes) 000260[1] 1763511055.068474 mapread 0x2fa000 thru 0x302422 (0x8423 bytes) 000262[0] 1763511055.080734 mapread 0x08a000 thru 0x095677 (0xb678 bytes) 000263[1] 1763511055.103732 trunc from 0x385190 to 0x17b026 (0x20a169 bytes) 000264[0] 1763511055.212005 mapwrite 0x89f000 thru 0x8a5b5a (0x6b5b bytes) 000266[1] 1763511055.387019 trunc from 0x8a5b5b to 0x7fbd69 (0xa9df1 bytes) 000267[1] 1763511055.435795 mapwrite 0x81f000 thru 0x81f93e (0x93f bytes) 000269[0] 1763511055.603756 write_OD 0x35c000 thru 0x36afff (0xf000 bytes) 000271[1] 1763511055.674689 trunc from 0x81f93f to 0x709013 (0x11692b bytes) 000276[1] 1763511055.736846 mapwrite 0x856000 thru 0x862a20 (0xca21 bytes) 000277[1] 1763511055.844567 trunc from 0x862a21 to 0x95b8ae (0xf8e8e bytes) 000279[1] 1763511055.885804 trunc from 0x95b8af to 0x56a0a5 (0x3f1809 bytes) 000281[1] 1763511055.907555 write 0x94f000 thru 0x95a882 (0xb883 bytes) 000284[1] 1763511055.933868 read 0x579000 thru 0x57b747 (0x2748 bytes) 000285[1] 1763511055.941563 read 0x534000 thru 0x53e386 (0xa387 bytes) 000288[0] 1763511055.991624 read_OD 0x51f000 thru 0x524fff (0x6000 bytes) 000290[1] 1763511056.034817 trunc from 0x95a883 to 0x8b19ba (0xa8ec8 bytes) 000293[0] 1763511056.062759 trunc from 0x8b19bb to 0x7e00b5 (0xd1905 bytes) 000295[0] 1763511056.113109 write_OD 0x3a1000 thru 0x3a9fff (0x9000 bytes) 000297[0] 1763511056.191357 write_OD 0x73a000 thru 0x740fff (0x7000 bytes) 000298[1] 1763511056.253303 write 0x68e000 thru 0x68f2f9 (0x12fa bytes) 000299[0] 1763511056.276145 write_OD 0x1d2000 thru 0x1ddfff (0xc000 bytes) 000301[0] 1763511056.369500 mapwrite 0x919000 thru 0x91e7f2 (0x57f3 bytes) 000302[0] 1763511056.547375 mapwrite 0x2dd000 thru 0x2ea574 (0xd575 bytes) 000306[1] 1763511056.656854 mapread 0x693000 thru 0x69ebb8 (0xbbb9 bytes) 000307[1] 1763511056.694628 trunc from 0x91e7f3 to 0x61c8a1 (0x301f51 bytes) 000309[0] 1763511056.725336 trunc from 0x61c8a2 to 0x167d37 (0x4b4b6a bytes) 000310[0] 1763511056.821843 mapread 0x008000 thru 0x0155a5 (0xd5a6 bytes) 000312[1] 1763511056.919373 read 0x009000 thru 0x00baf6 (0x2af7 bytes) 000314[1] 1763511057.022294 mapread 0x0ce000 thru 0x0dcd4d (0xed4e bytes) 000317[1] 1763511057.023459 write 0x8a3000 thru 0x8a5dfc (0x2dfd bytes) 000320[0] 1763511057.131483 mapwrite 0x34b000 thru 0x34db5c (0x2b5d bytes) 000322[1] 1763511057.242798 mapread 0x87f000 thru 0x88d901 (0xe902 bytes) 000326[0] 1763511057.416350 read_OD 0x644000 thru 0x647fff (0x4000 bytes) 000327[0] 1763511057.429672 mapread 0x109000 thru 0x10a323 (0x1324 bytes) 000328[1] 1763511057.446855 trunc from 0x8a5dfd to 0x152033 (0x753dc9 bytes) 000329[0] 1763511057.513270 mapread 0x136000 thru 0x143862 (0xd863 bytes) 000330[0] 1763511057.585621 write_OD 0x72d000 thru 0x72ffff (0x3000 bytes) 000331[1] 1763511057.642668 read 0x65e000 thru 0x668f5c (0xaf5d bytes) 000333[1] 1763511057.761663 read 0x25a000 thru 0x264794 (0xa795 bytes) 000336[1] 1763511057.763995 trunc from 0x730000 to 0x3cb0a0 (0x364f5f bytes) 000338[0] 1763511057.825336 mapread 0x37b000 thru 0x389921 (0xe922 bytes) 000339[1] 1763511057.891988 mapwrite 0x7cd000 thru 0x7cdc17 (0xc18 bytes) 000340[0] 1763511057.995446 read_OD 0x348000 thru 0x34efff (0x7000 bytes) 000343[0] 1763511058.038207 read_OD 0x591000 thru 0x598fff (0x8000 bytes) 000344[0] 1763511058.062809 trunc from 0x7cdc18 to 0x163a3d (0x66a1da bytes) 000346[0] 1763511058.126019 mapread 0x02c000 thru 0x02e6fc (0x26fd bytes) 000347[0] 1763511058.163954 mapread 0x122000 thru 0x12e623 (0xc624 bytes) 000348[1] 1763511058.181690 mapwrite 0x8e1000 thru 0x8e4eb8 (0x3eb9 bytes) 000350[0] 1763511058.329672 trunc from 0x8e4eb9 to 0x8b743f (0x2da79 bytes) 000351[1] 1763511058.356032 trunc from 0x8b7440 to 0x858785 (0x5ecba bytes) 000352[0] 1763511058.391144 read_OD 0x0ba000 thru 0x0c1fff (0x8000 bytes) 000354[0] 1763511058.436196 read_OD 0x5a7000 thru 0x5abfff (0x5000 bytes) 000355[1] 1763511058.446938 mapread 0x02c000 thru 0x033e90 (0x7e91 bytes) 000357[0] 1763511058.467713 write_OD 0x21a000 thru 0x225fff (0xc000 bytes) 000358[1] 1763511058.568788 mapwrite 0x440000 thru 0x44fc5b (0xfc5c bytes) 000359[1] 1763511058.690608 mapread 0x43c000 thru 0x4467de (0xa7df bytes) 000361[0] 1763511058.691953 read_OD 0x32f000 thru 0x32ffff (0x1000 bytes) 000362[1] 1763511058.715632 mapread 0x36a000 thru 0x37752f (0xd530 bytes) 000363[0] 1763511058.747547 read_OD 0x2dc000 thru 0x2e9fff (0xe000 bytes) 000364[1] 1763511058.757128 trunc from 0x858786 to 0x613df8 (0x24498d bytes) 000365[1] 1763511058.806860 trunc from 0x613df9 to 0x326486 (0x2ed972 bytes) 000366[1] 1763511058.839283 mapread 0x018000 thru 0x022d53 (0xad54 bytes) 000367[0] 1763511058.911406 mapread 0x305000 thru 0x30cab7 (0x7ab8 bytes) 000368[0] 1763511058.953957 read_OD 0x0f7000 thru 0x0f8fff (0x2000 bytes) 000369[0] 1763511058.963460 mapread 0x032000 thru 0x033c3e (0x1c3f bytes) 000373[1] 1763511059.019760 trunc from 0x326487 to 0x72b8b2 (0x40542c bytes) 000374[0] 1763511059.072342 mapwrite 0x126000 thru 0x1302db (0xa2dc bytes) 000375[0] 1763511059.259987 trunc from 0x72b8b3 to 0x5c8396 (0x16351c bytes) 000376[0] 1763511059.283166 trunc from 0x5c8397 to 0x35d090 (0x26b306 bytes) 000378[0] 1763511059.368532 mapread 0x131000 thru 0x137d28 (0x6d29 bytes) 000380[1] 1763511059.410492 trunc from 0x35d091 to 0x5fb331 (0x29e2a1 bytes) 000381[0] 1763511059.439559 read_OD 0x055000 thru 0x05efff (0xa000 bytes) 000384[0] 1763511059.492039 mapwrite 0x010000 thru 0x01d488 (0xd489 bytes) 000385[1] 1763511059.654121 mapread 0x291000 thru 0x2a0824 (0xf825 bytes) 000386[1] 1763511059.718786 read 0x1e6000 thru 0x1e8776 (0x2777 bytes) 000387[0] 1763511059.734991 write_OD 0x6c3000 thru 0x6d0fff (0xe000 bytes) 000388[0] 1763511059.811031 mapread 0x1bb000 thru 0x1c0c25 (0x5c26 bytes) 000389[1] 1763511059.847849 mapread 0x0d4000 thru 0x0e0e37 (0xce38 bytes) 000391[1] 1763511059.879951 trunc from 0x6d1000 to 0x2d6ba4 (0x3fa45b bytes) 000392[0] 1763511059.927441 trunc from 0x2d6ba5 to 0x33c276 (0x656d2 bytes) 000393[0] 1763511059.988463 write_OD 0x012000 thru 0x01afff (0x9000 bytes) 000394[1] 1763511060.056326 mapread 0x24a000 thru 0x24ef9c (0x4f9d bytes) 000396[0] 1763511060.088719 mapwrite 0x973000 thru 0x981fb5 (0xefb6 bytes) 000398[1] 1763511060.291344 mapread 0x42e000 thru 0x4343e2 (0x63e3 bytes) 000399[1] 1763511060.340511 mapwrite 0x6ca000 thru 0x6d6c1a (0xcc1b bytes) 000400[0] 1763511060.449100 read_OD 0x427000 thru 0x427fff (0x1000 bytes) 000403[1] 1763511060.468963 trunc from 0x981fb6 to 0x183e31 (0x7fe184 bytes) 000404[1] 1763511060.506084 mapwrite 0x322000 thru 0x3245c8 (0x25c9 bytes) 000406[1] 1763511060.621934 mapwrite 0x4e6000 thru 0x4ed633 (0x7634 bytes) 000407[0] 1763511060.720315 mapread 0x3f3000 thru 0x3ff126 (0xc127 bytes) 000408[1] 1763511060.776416 read 0x40f000 thru 0x41b2e0 (0xc2e1 bytes) 000409[1] 1763511060.783477 write 0x24d000 thru 0x25ce5e (0xfe5f bytes) 000410[1] 1763511060.803042 write 0x6ac000 thru 0x6ad49f (0x14a0 bytes) 000411[1] 1763511060.804549 mapwrite 0x637000 thru 0x64336b (0xc36c bytes) 000415[1] 1763511060.917503 write 0x687000 thru 0x68d09c (0x609d bytes) 000417[0] 1763511060.952988 mapread 0x568000 thru 0x575e70 (0xde71 bytes) 000418[0] 1763511061.043628 trunc from 0x6ad4a0 to 0x4431e9 (0x26a2b6 bytes) 000421[1] 1763511061.123591 trunc from 0x4431ea to 0x09c9d8 (0x3a6811 bytes) 000423[1] 1763511061.202753 mapread 0x07b000 thru 0x08a2af (0xf2b0 bytes) 000426[1] 1763511061.262107 read 0x099000 thru 0x09b837 (0x2838 bytes) 000427[0] 1763511061.284332 read_OD 0x001000 thru 0x002fff (0x2000 bytes) 000430[1] 1763511061.320248 mapwrite 0x111000 thru 0x1152e3 (0x42e4 bytes) 000431[1] 1763511061.440788 write 0x855000 thru 0x85f8f8 (0xa8f9 bytes) 000434[1] 1763511061.490278 write 0x3c0000 thru 0x3cb54d (0xb54e bytes) 000438[1] 1763511061.599721 mapread 0x1e7000 thru 0x1ebc87 (0x4c88 bytes) 000439[0] 1763511061.629043 mapwrite 0x31f000 thru 0x323867 (0x4868 bytes) 000441[0] 1763511061.829564 write_OD 0x29d000 thru 0x2a0fff (0x4000 bytes) 000442[1] 1763511061.889060 mapread 0x60c000 thru 0x60f8aa (0x38ab bytes) 000443[1] 1763511061.996252 trunc from 0x85f8f9 to 0x140301 (0x71f5f7 bytes) 000450[0] 1763511062.138041 read_OD 0x005000 thru 0x00efff (0xa000 bytes) 000453[1] 1763511062.169244 write 0x6cb000 thru 0x6d032b (0x532c bytes) 000454[1] 1763511062.184537 read 0x527000 thru 0x52d4ba (0x64bb bytes) 000456[1] 1763511062.228413 trunc from 0x6d032c to 0x141caf (0x58e67c bytes) 000459[0] 1763511062.301290 read_OD 0x0b6000 thru 0x0c0fff (0xb000 bytes) 000461[1] 1763511062.370787 read 0x024000 thru 0x03298e (0xe98f bytes) 000462[1] 1763511062.398738 mapread 0x063000 thru 0x071c3d (0xec3e bytes) 000463[0] 1763511062.401398 mapwrite 0x90a000 thru 0x90fcf0 (0x5cf1 bytes) 000464[1] 1763511062.649373 mapread 0x350000 thru 0x356539 (0x653a bytes) 000466[0] 1763511062.765462 trunc from 0x90fcf1 to 0x74532b (0x1ca9c5 bytes) 000470[0] 1763511062.874781 write_OD 0x944000 thru 0x94bfff (0x8000 bytes) 000473[1] 1763511062.950496 mapwrite 0x0ff000 thru 0x10b3ea (0xc3eb bytes) 000477[0] 1763511063.188207 mapread 0x025000 thru 0x032c23 (0xdc24 bytes) 000478[0] 1763511063.296243 trunc from 0x94c000 to 0x43b27c (0x510d83 bytes) 000479[0] 1763511063.338740 mapwrite 0x950000 thru 0x95c141 (0xc142 bytes) 000485[0] 1763511063.518334 read_OD 0x1a7000 thru 0x1a8fff (0x2000 bytes) 000486[0] 1763511063.539076 write_OD 0x2c0000 thru 0x2ccfff (0xd000 bytes) 000490[0] 1763511063.626515 trunc from 0x95c142 to 0x13cf2d (0x81f214 bytes) 000492[1] 1763511063.718097 write 0x23e000 thru 0x23e3ee (0x3ef bytes) 000493[1] 1763511063.774294 trunc from 0x23e3ef to 0x5272d2 (0x2e8ee4 bytes) 000494[0] 1763511063.812383 mapwrite 0x56f000 thru 0x57d2bb (0xe2bc bytes) 000496[1] 1763511063.971492 mapread 0x570000 thru 0x575b06 (0x5b07 bytes) 000497[1] 1763511064.011110 write 0x593000 thru 0x5a0db4 (0xddb5 bytes) 000498[0] 1763511064.057163 read_OD 0x492000 thru 0x492fff (0x1000 bytes) 000499[0] 1763511064.109546 trunc from 0x5a0db5 to 0x366732 (0x23a682 bytes) All operations completed A-OK! PASS 16a (47s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 19:11:08 (1763511068) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.255251 s, 32.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.27641 s, 30.3 MB/s Chance of close/open is 1 in 50 Seed set to 1070 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1763511070.943108 write 0x463ff9 thru 0x464a22 (0xa2a bytes) 000004[0] 1763511071.003259 mapread 0x07d241 thru 0x07f202 (0x1fc2 bytes) 000008[1] 1763511071.063999 mapread 0x20d796 thru 0x20f5a0 (0x1e0b bytes) 000012[0] 1763511071.198636 mapwrite 0x6131b4 thru 0x614bc5 (0x1a12 bytes) truncating to largest ever: 0x4e0589 000013[1] 1763511071.364099 trunc from 0x614bc6 to 0x4e0588 (0x13463d bytes) 000015[0] 1763511071.429889 mapread 0x133c65 thru 0x13409f (0x43b bytes) 000018[1] 1763511071.490609 read 0x4a6534 thru 0x4a6a59 (0x526 bytes) 000019[1] 1763511071.496199 mapread 0x2ee52d thru 0x2eefdf (0xab3 bytes) 000024[1] 1763511071.544367 mapread 0x1730de thru 0x174d56 (0x1c79 bytes) 000026[0] 1763511071.577779 mapwrite 0x656caf thru 0x657e35 (0x1187 bytes) 000030[0] 1763511071.780969 read 0x3f43ee thru 0x3f5ba4 (0x17b7 bytes) 000031[1] 1763511071.809657 mapread 0x645a8d thru 0x6478a9 (0x1e1d bytes) 000032[0] 1763511071.870814 mapwrite 0x89d893 thru 0x89f0fe (0x186c bytes) 000033[0] 1763511072.031079 mapread 0x1083c7 thru 0x10a070 (0x1caa bytes) 000034[1] 1763511072.075005 trunc from 0x89f0ff to 0x29d42c (0x601cd2 bytes) 000037[1] 1763511072.179467 trunc from 0x29d42d to 0x447daf (0x1aa983 bytes) 000038[0] 1763511072.199943 mapread 0x1cae57 thru 0x1ccaf7 (0x1ca1 bytes) 000041[1] 1763511072.310349 mapwrite 0x8b6edd thru 0x8b75af (0x6d3 bytes) 000043[1] 1763511072.443574 mapwrite 0x5b1965 thru 0x5b200c (0x6a8 bytes) 000044[1] 1763511072.508354 mapwrite 0x05c0ca thru 0x05c722 (0x659 bytes) 000045[1] 1763511072.580165 write 0x597c7a thru 0x599897 (0x1c1e bytes) 000046[1] 1763511072.593100 read 0x6dacd1 thru 0x6db537 (0x867 bytes) 000048[0] 1763511072.655841 write 0x8e06fc thru 0x8e0996 (0x29b bytes) truncating to largest ever: 0x7cdd17 000049[1] 1763511072.684794 trunc from 0x8e0997 to 0x7cdd16 (0x112c80 bytes) 000050[0] 1763511072.726809 mapwrite 0x65a995 thru 0x65c541 (0x1bad bytes) 000051[0] 1763511072.850506 mapread 0x72d0ec thru 0x72e0db (0xff0 bytes) 000052[1] 1763511072.897308 read 0x358cca thru 0x35a5f8 (0x192f bytes) 000054[1] 1763511072.950423 write 0x6bda57 thru 0x6beb32 (0x10dc bytes) 000055[0] 1763511072.977997 read 0x5ba3d1 thru 0x5bc0d9 (0x1d09 bytes) 000060[1] 1763511073.041674 write 0x8469b7 thru 0x8479e2 (0x102c bytes) 000061[0] 1763511073.044655 read 0x3520a8 thru 0x35407d (0x1fd6 bytes) truncating to largest ever: 0x925e07 000063[0] 1763511073.072058 trunc from 0x8479e3 to 0x925e06 (0xde424 bytes) 000064[1] 1763511073.142071 mapread 0x0557fb thru 0x05665b (0xe61 bytes) 000065[0] 1763511073.204493 write 0x8d85bb thru 0x8da416 (0x1e5c bytes) 000066[1] 1763511073.243175 trunc from 0x925e07 to 0x725b44 (0x2002c2 bytes) 000068[0] 1763511073.306625 read 0x1f40d9 thru 0x1f4a24 (0x94c bytes) 000070[1] 1763511073.335842 mapread 0x58ce69 thru 0x58d501 (0x699 bytes) 000071[0] 1763511073.373168 mapread 0x220d28 thru 0x2223fd (0x16d6 bytes) 000072[0] 1763511073.415284 write 0x51fecd thru 0x520fa6 (0x10da bytes) 000074[1] 1763511073.437875 mapread 0x30563c thru 0x305da0 (0x765 bytes) 000076[0] 1763511073.449400 trunc from 0x725b45 to 0x46a77d (0x2bb3c7 bytes) 000079[0] 1763511073.557823 trunc from 0x46a77e to 0x04cf14 (0x41d869 bytes) 000082[1] 1763511073.642495 mapwrite 0x402146 thru 0x403d68 (0x1c23 bytes) 000088[1] 1763511073.850566 trunc from 0x403d69 to 0x2a1547 (0x162821 bytes) 000089[0] 1763511073.924414 write 0x94f7a1 thru 0x9501d9 (0xa39 bytes) 000090[0] 1763511073.972440 mapwrite 0x92c380 thru 0x92cec0 (0xb41 bytes) 000091[0] 1763511074.072013 read 0x03e8a6 thru 0x03fb20 (0x127b bytes) truncating to largest ever: 0x9a7699 000094[0] 1763511074.145782 trunc from 0x9501da to 0x9a7698 (0x574bf bytes) 000099[0] 1763511074.312410 mapread 0x77365d thru 0x774cde (0x1682 bytes) 000102[1] 1763511074.330456 write 0x6b3ceb thru 0x6b507b (0x1391 bytes) 000104[0] 1763511074.394616 mapwrite 0x4b9715 thru 0x4baac4 (0x13b0 bytes) 000108[1] 1763511074.512777 mapread 0x3a7452 thru 0x3a781b (0x3ca bytes) 000109[1] 1763511074.569529 trunc from 0x9a7699 to 0x3f3ef9 (0x5b379f bytes) 000117[1] 1763511074.664553 mapwrite 0x1a2206 thru 0x1a3d42 (0x1b3d bytes) 000118[1] 1763511074.745379 read 0x3a0177 thru 0x3a0cd7 (0xb61 bytes) 000119[0] 1763511074.842565 write 0x365cdd thru 0x3678d5 (0x1bf9 bytes) 000122[1] 1763511074.928148 read 0x1c2eaa thru 0x1c3afc (0xc53 bytes) 000124[0] 1763511075.023535 read 0x0e7e1b thru 0x0e92d8 (0x14be bytes) truncating to largest ever: 0x9d8879 000125[0] 1763511075.095343 trunc from 0x3f3efa to 0x9d8878 (0x5e497f bytes) 000126[0] 1763511075.168005 read 0x760963 thru 0x76243c (0x1ada bytes) 000129[0] 1763511075.214787 write 0x830c4c thru 0x83182d (0xbe2 bytes) 000130[0] 1763511075.254953 write 0x4003ff thru 0x400a28 (0x62a bytes) 000131[1] 1763511075.292605 trunc from 0x9d8879 to 0x2e25e0 (0x6f6298 bytes) 000132[0] 1763511075.350084 mapread 0x123102 thru 0x124b54 (0x1a53 bytes) 000133[1] 1763511075.383787 read 0x2c7f9f thru 0x2c9a37 (0x1a99 bytes) 000135[1] 1763511075.470879 read 0x0c575f thru 0x0c6fa7 (0x1849 bytes) 000136[0] 1763511075.486367 write 0x6b0f3b thru 0x6b17f3 (0x8b9 bytes) 000139[1] 1763511075.511212 trunc from 0x6b17f4 to 0x170784 (0x54106f bytes) 000144[1] 1763511075.603362 read 0x1204fc thru 0x121a80 (0x1585 bytes) 000147[0] 1763511075.645389 read 0x1017e3 thru 0x1035c8 (0x1de6 bytes) 000150[1] 1763511075.754877 read 0x081f94 thru 0x0838b8 (0x1925 bytes) 000152[0] 1763511075.800518 read 0x11f9cd thru 0x120050 (0x684 bytes) 000153[0] 1763511075.801042 read 0x141d5c thru 0x142b99 (0xe3e bytes) 000156[0] 1763511075.833864 write 0x15d825 thru 0x15db12 (0x2ee bytes) truncating to largest ever: 0x9e816c 000159[0] 1763511075.950065 trunc from 0x170785 to 0x9e816b (0x8779e7 bytes) 000162[1] 1763511076.045500 mapread 0x290b83 thru 0x2915f5 (0xa73 bytes) 000164[0] 1763511076.194853 write 0x24cd35 thru 0x24e06f (0x133b bytes) 000165[1] 1763511076.282121 write 0x2e086c thru 0x2e137a (0xb0f bytes) 000166[0] 1763511076.349318 read 0x5e1224 thru 0x5e28e3 (0x16c0 bytes) 000168[1] 1763511076.391156 read 0x12a630 thru 0x12bc24 (0x15f5 bytes) 000171[1] 1763511076.440487 write 0x67088e thru 0x6723f0 (0x1b63 bytes) 000172[1] 1763511076.463689 mapwrite 0x81e9fd thru 0x8203a4 (0x19a8 bytes) 000177[1] 1763511076.549680 trunc from 0x9e816c to 0x10b852 (0x8dc919 bytes) 000178[0] 1763511076.588449 read 0x0cca93 thru 0x0cd69b (0xc09 bytes) 000182[0] 1763511076.660311 mapread 0x01d163 thru 0x01e1aa (0x1048 bytes) 000186[0] 1763511076.738128 mapwrite 0x5ea89d thru 0x5eaedb (0x63f bytes) 000188[1] 1763511076.849964 mapread 0x39abcf thru 0x39acb6 (0xe8 bytes) 000189[0] 1763511076.879769 write 0x60ecb1 thru 0x60fbfd (0xf4d bytes) 000190[0] 1763511076.880883 write 0x6a157a thru 0x6a1b96 (0x61d bytes) 000191[0] 1763511076.894384 read 0x514ed0 thru 0x515bcb (0xcfc bytes) 000192[0] 1763511076.916476 read 0x3fd094 thru 0x3fd725 (0x692 bytes) 000193[0] 1763511076.946059 trunc from 0x6a1b97 to 0x86da95 (0x1cbeff bytes) 000195[1] 1763511077.012177 mapread 0x51c8b7 thru 0x51d3a6 (0xaf0 bytes) 000197[0] 1763511077.071555 write 0x2adccc thru 0x2afb98 (0x1ecd bytes) 000198[0] 1763511077.090959 mapwrite 0x99ff7e thru 0x9a1e30 (0x1eb3 bytes) 000199[0] 1763511077.207077 write 0x6cc41a thru 0x6cdf21 (0x1b08 bytes) 000201[0] 1763511077.226428 read 0x292d32 thru 0x292f98 (0x267 bytes) 000204[1] 1763511077.264071 mapread 0x62ac5b thru 0x62b182 (0x528 bytes) 000205[1] 1763511077.323510 write 0x5230d4 thru 0x5235a6 (0x4d3 bytes) 000207[0] 1763511077.396321 write 0x15b8ad thru 0x15ccac (0x1400 bytes) 000208[1] 1763511077.428882 trunc from 0x9a1e31 to 0x18bbf3 (0x81623d bytes) 000210[1] 1763511077.511195 write 0x94d2ce thru 0x94e94e (0x1681 bytes) 000211[0] 1763511077.518049 mapwrite 0x6499ef thru 0x64a6c6 (0xcd8 bytes) 000213[0] 1763511077.662148 mapread 0x42c12d thru 0x42c3a9 (0x27d bytes) 000214[0] 1763511077.725433 read 0x521756 thru 0x521dfb (0x6a6 bytes) 000216[0] 1763511077.804872 write 0x319436 thru 0x31a35b (0xf26 bytes) 000220[1] 1763511077.923151 trunc from 0x94e94f to 0x63ba28 (0x312f26 bytes) 000223[1] 1763511077.979043 read 0x1ce95b thru 0x1d0105 (0x17ab bytes) 000226[0] 1763511078.064980 mapread 0x3c761f thru 0x3c8752 (0x1134 bytes) 000227[1] 1763511078.144890 mapread 0x172a05 thru 0x172ddd (0x3d9 bytes) 000228[0] 1763511078.180588 mapwrite 0x1b670b thru 0x1b84f1 (0x1de7 bytes) 000229[0] 1763511078.319129 trunc from 0x63ba29 to 0x16d3be (0x4ce66a bytes) 000235[0] 1763511078.409361 read 0x0f4680 thru 0x0f5f50 (0x18d1 bytes) 000237[1] 1763511078.501343 mapwrite 0x536822 thru 0x53773b (0xf1a bytes) 000238[0] 1763511078.618068 read 0x2482cf thru 0x248a9f (0x7d1 bytes) 000240[1] 1763511078.718759 mapwrite 0x4c2019 thru 0x4c34b1 (0x1499 bytes) 000241[1] 1763511078.792787 mapread 0x45617d thru 0x4572de (0x1162 bytes) 000242[0] 1763511078.816129 write 0x39b7bf thru 0x39d5ba (0x1dfc bytes) 000244[0] 1763511078.879434 write 0x9eabc3 thru 0x9eb79f (0xbdd bytes) 000245[1] 1763511078.898946 write 0x5ad1e2 thru 0x5af08d (0x1eac bytes) 000246[0] 1763511078.919603 mapwrite 0x755a38 thru 0x755e84 (0x44d bytes) 000250[1] 1763511079.084823 mapread 0x448e71 thru 0x44adc0 (0x1f50 bytes) 000251[1] 1763511079.171278 write 0x470ac8 thru 0x470b8c (0xc5 bytes) 000253[1] 1763511079.256632 trunc from 0x9eb7a0 to 0x371e41 (0x67995e bytes) 000255[0] 1763511079.317672 mapwrite 0x078044 thru 0x07954e (0x150b bytes) 000257[1] 1763511079.467201 mapread 0x2f587e thru 0x2f65e6 (0xd69 bytes) 000258[1] 1763511079.519239 write 0x68c30f thru 0x68d053 (0xd45 bytes) 000259[1] 1763511079.521644 trunc from 0x68d054 to 0x5a8a05 (0xe464e bytes) 000264[0] 1763511079.606282 trunc from 0x5a8a06 to 0x19ce33 (0x40bbd2 bytes) 000265[1] 1763511079.678815 write 0x11d922 thru 0x11f8dd (0x1fbc bytes) 000266[0] 1763511079.729110 mapread 0x05d7dc thru 0x05f17e (0x19a3 bytes) 000267[1] 1763511079.858692 write 0x16ffcc thru 0x1704bc (0x4f1 bytes) 000268[0] 1763511079.963522 read 0x0a468b thru 0x0a5999 (0x130f bytes) 000274[0] 1763511080.064524 read 0x11de1f thru 0x11fcd7 (0x1eb9 bytes) 000275[0] 1763511080.065153 write 0x57c0d7 thru 0x57dca5 (0x1bcf bytes) 000279[0] 1763511080.116158 write 0x118c5a thru 0x11a1d9 (0x1580 bytes) 000280[0] 1763511080.208369 trunc from 0x57dca6 to 0x4cc757 (0xb154e bytes) 000281[0] 1763511080.232589 mapwrite 0x85ddda thru 0x85e979 (0xba0 bytes) 000282[1] 1763511080.438776 mapwrite 0x0f5ec2 thru 0x0f6807 (0x946 bytes) 000283[1] 1763511080.533216 mapwrite 0x698d28 thru 0x698dac (0x85 bytes) 000288[0] 1763511080.644217 write 0x0bf886 thru 0x0c0ca5 (0x1420 bytes) 000289[1] 1763511080.709307 write 0x034c0c thru 0x0356da (0xacf bytes) 000290[1] 1763511080.760964 mapread 0x608241 thru 0x609941 (0x1701 bytes) 000291[1] 1763511080.782654 trunc from 0x85e97a to 0x4cad54 (0x393c25 bytes) 000292[1] 1763511080.824731 trunc from 0x4cad55 to 0x1ec350 (0x2dea04 bytes) 000293[1] 1763511080.851744 mapread 0x18c379 thru 0x18d3af (0x1037 bytes) 000294[0] 1763511080.878641 trunc from 0x1ec351 to 0x3b8a8c (0x1cc73c bytes) 000295[1] 1763511081.010242 mapread 0x2a8cd4 thru 0x2aaca9 (0x1fd6 bytes) 000296[1] 1763511081.070112 write 0x5692e0 thru 0x56ad97 (0x1ab8 bytes) 000303[1] 1763511081.170818 read 0x31af2b thru 0x31b2ee (0x3c4 bytes) 000305[0] 1763511081.209965 write 0x78fbdc thru 0x791494 (0x18b9 bytes) 000306[0] 1763511081.264757 mapread 0x5a88c6 thru 0x5a88d5 (0x10 bytes) 000308[0] 1763511081.272843 mapwrite 0x001573 thru 0x002cad (0x173b bytes) 000309[0] 1763511081.389208 mapread 0x4b3169 thru 0x4b3ae9 (0x981 bytes) 000310[0] 1763511081.420598 read 0x00e040 thru 0x00f288 (0x1249 bytes) 000314[1] 1763511081.449952 trunc from 0x791495 to 0x38cb86 (0x40490e bytes) 000315[0] 1763511081.533630 read 0x1ddca2 thru 0x1ddd9c (0xfb bytes) 000316[1] 1763511081.607998 trunc from 0x38cb87 to 0x1ec720 (0x1a0466 bytes) 000318[0] 1763511081.696692 mapread 0x1736d0 thru 0x174590 (0xec1 bytes) 000320[1] 1763511081.761432 mapwrite 0x7803f3 thru 0x780885 (0x493 bytes) 000325[1] 1763511081.920005 trunc from 0x780886 to 0x16242d (0x61e458 bytes) 000327[0] 1763511081.969201 mapread 0x13489f thru 0x134bac (0x30e bytes) 000329[1] 1763511082.015174 mapwrite 0x0524c0 thru 0x052ba1 (0x6e2 bytes) 000332[0] 1763511082.104197 write 0x482e73 thru 0x48425b (0x13e9 bytes) 000334[0] 1763511082.166635 read 0x13551f thru 0x1366e8 (0x11ca bytes) 000336[0] 1763511082.208038 write 0x3e86cb thru 0x3e979f (0x10d5 bytes) 000338[0] 1763511082.234192 read 0x2a2400 thru 0x2a3e4a (0x1a4b bytes) 000339[1] 1763511082.243665 write 0x0812c7 thru 0x081fdd (0xd17 bytes) 000341[1] 1763511082.293275 write 0x0cfa52 thru 0x0d0c45 (0x11f4 bytes) 000344[1] 1763511082.406878 read 0x3ba307 thru 0x3ba777 (0x471 bytes) 000345[0] 1763511082.441522 trunc from 0x48425c to 0x1b0d06 (0x2d3555 bytes) 000346[1] 1763511082.493930 mapwrite 0x8e9403 thru 0x8ea781 (0x137f bytes) 000348[0] 1763511082.729299 trunc from 0x8ea782 to 0x689009 (0x261778 bytes) 000351[0] 1763511082.858603 mapwrite 0x1524a6 thru 0x153f29 (0x1a84 bytes) 000354[0] 1763511082.953582 trunc from 0x68900a to 0x7b0ec1 (0x127eb8 bytes) 000357[0] 1763511083.031799 trunc from 0x7b0ec2 to 0x424842 (0x38c67f bytes) 000362[0] 1763511083.108286 write 0x3480f5 thru 0x349634 (0x1540 bytes) 000364[0] 1763511083.118590 mapwrite 0x5f519a thru 0x5f6ed9 (0x1d40 bytes) 000368[1] 1763511083.322772 mapwrite 0x622383 thru 0x622a28 (0x6a6 bytes) 000369[1] 1763511083.487182 trunc from 0x622a29 to 0x7e76fb (0x1c4cd3 bytes) 000372[0] 1763511083.546641 trunc from 0x7e76fc to 0x3dbdc7 (0x40b934 bytes) 000375[0] 1763511083.684055 read 0x0dc2b9 thru 0x0ddbd0 (0x1918 bytes) 000376[1] 1763511083.787707 read 0x319c92 thru 0x31bb78 (0x1ee7 bytes) 000377[0] 1763511083.906475 trunc from 0x3dbdc8 to 0x9b1737 (0x5d5970 bytes) 000379[1] 1763511083.991821 write 0x43e25c thru 0x43e726 (0x4cb bytes) 000381[1] 1763511084.060452 write 0x50b028 thru 0x50cc46 (0x1c1f bytes) 000382[0] 1763511084.092529 write 0x8e3683 thru 0x8e41bf (0xb3d bytes) 000383[1] 1763511084.123320 write 0x3ccdf7 thru 0x3cd424 (0x62e bytes) 000384[1] 1763511084.166568 mapwrite 0x986fd3 thru 0x988da1 (0x1dcf bytes) 000387[0] 1763511084.352784 mapwrite 0x26359a thru 0x264ab1 (0x1518 bytes) 000388[0] 1763511084.455388 read 0x014fe4 thru 0x01678d (0x17aa bytes) 000389[1] 1763511084.475492 write 0x602157 thru 0x602482 (0x32c bytes) 000390[1] 1763511084.489605 write 0x49777b thru 0x49909b (0x1921 bytes) 000391[0] 1763511084.505862 trunc from 0x9b1738 to 0x7b988e (0x1f7ea9 bytes) 000392[1] 1763511084.584215 write 0x2d0aad thru 0x2d15cc (0xb20 bytes) 000394[1] 1763511084.650944 mapread 0x40c01d thru 0x40d147 (0x112b bytes) 000397[0] 1763511084.738712 read 0x51a747 thru 0x51c36a (0x1c24 bytes) 000399[0] 1763511084.787151 trunc from 0x7b988f to 0x87071d (0xb6e8f bytes) 000401[0] 1763511084.881870 trunc from 0x87071e to 0x5d9ecf (0x29684e bytes) 000404[0] 1763511084.945579 mapread 0x02c45f thru 0x02c81b (0x3bd bytes) 000405[1] 1763511084.967504 read 0x3db9ca thru 0x3dc2e2 (0x919 bytes) 000408[1] 1763511085.006005 write 0x231b4f thru 0x23305e (0x1510 bytes) 000409[1] 1763511085.045313 write 0x989084 thru 0x989d26 (0xca3 bytes) 000411[1] 1763511085.087795 mapwrite 0x4fff26 thru 0x5016ac (0x1787 bytes) 000412[1] 1763511085.187949 mapwrite 0x984424 thru 0x98566b (0x1248 bytes) 000414[1] 1763511085.248132 mapwrite 0x6b964e thru 0x6ba610 (0xfc3 bytes) 000416[0] 1763511085.296573 write 0x088421 thru 0x089f27 (0x1b07 bytes) 000417[0] 1763511085.342383 trunc from 0x989d27 to 0x6f5cb0 (0x294076 bytes) 000418[1] 1763511085.375443 read 0x361ce8 thru 0x362bc9 (0xee2 bytes) 000427[1] 1763511085.509511 mapread 0x2e47bb thru 0x2e6662 (0x1ea8 bytes) 000431[0] 1763511085.566414 write 0x770014 thru 0x7707be (0x7ab bytes) 000432[1] 1763511085.581693 mapread 0x1b6d97 thru 0x1b84c5 (0x172f bytes) 000437[1] 1763511085.711277 write 0x702d17 thru 0x704420 (0x170a bytes) 000439[1] 1763511085.775841 trunc from 0x7707bf to 0x6f1ceb (0x7ead3 bytes) 000441[0] 1763511085.865183 write 0x08aa0f thru 0x08afbf (0x5b1 bytes) 000444[1] 1763511085.927083 mapread 0x4c8b78 thru 0x4ca2cc (0x1755 bytes) 000445[1] 1763511085.982548 write 0x8abc13 thru 0x8ace7a (0x1268 bytes) 000446[0] 1763511086.034425 mapread 0x268bff thru 0x268c7f (0x81 bytes) 000448[0] 1763511086.110027 write 0x08af13 thru 0x08c2fd (0x13eb bytes) 000450[0] 1763511086.165013 trunc from 0x8ace7b to 0x488a92 (0x4243e8 bytes) 000452[1] 1763511086.231513 trunc from 0x488a93 to 0x60beb3 (0x183421 bytes) 000453[0] 1763511086.289469 read 0x4cbe12 thru 0x4cd15a (0x1349 bytes) 000454[1] 1763511086.328241 mapread 0x18c3a5 thru 0x18e3a3 (0x1fff bytes) 000457[0] 1763511086.366134 write 0x1a2541 thru 0x1a337e (0xe3e bytes) 000459[0] 1763511086.388974 mapread 0x23c8d4 thru 0x23dabc (0x11e9 bytes) 000462[1] 1763511086.415252 read 0x1d5b8a thru 0x1d5bdf (0x56 bytes) 000463[0] 1763511086.458688 mapread 0x601a45 thru 0x602087 (0x643 bytes) 000464[0] 1763511086.469386 write 0x2997d5 thru 0x29aa5f (0x128b bytes) 000465[1] 1763511086.487247 mapread 0x005fc6 thru 0x0063c4 (0x3ff bytes) 000467[0] 1763511086.532069 mapread 0x365093 thru 0x365c3f (0xbad bytes) 000468[0] 1763511086.542211 read 0x2f354f thru 0x2f4c5a (0x170c bytes) 000470[1] 1763511086.549688 mapread 0x2d73ec thru 0x2d8c7b (0x1890 bytes) 000473[0] 1763511086.563283 write 0x683628 thru 0x684540 (0xf19 bytes) 000478[0] 1763511086.596756 write 0x62d094 thru 0x62ecb9 (0x1c26 bytes) 000479[1] 1763511086.608775 mapread 0x2fa207 thru 0x2fbb0e (0x1908 bytes) 000481[0] 1763511086.627457 trunc from 0x684541 to 0x764e69 (0xe0929 bytes) 000484[0] 1763511086.710179 mapwrite 0x39784b thru 0x39951c (0x1cd2 bytes) 000488[0] 1763511086.873089 mapread 0x612e25 thru 0x6131a7 (0x383 bytes) 000489[0] 1763511086.884474 trunc from 0x764e6a to 0x1ca225 (0x59ac44 bytes) 000496[1] 1763511087.007855 read 0x10bc0d thru 0x10da55 (0x1e49 bytes) 000500[1] 1763511087.130460 read 0x124a74 thru 0x126795 (0x1d22 bytes) All operations completed A-OK! PASS 16b (23s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 19:11:31 (1763511091) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 19:11:34 (1763511094) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.355862 s, 23.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.26651 s, 31.5 MB/s ldlm.namespaces.MGC192.168.201.138@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9af785e88800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9af798626000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9af785e88800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9af785e88800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6634 s, 9.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.69713 s, 38.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.18699 s, 47.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43013 s, 19.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.24597 s, 16.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.41555 s, 43.4 MB/s PASS 16d (42s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 19:12:16 (1763511136) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.164061 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091913 s, 11.4 MB/s PASS 16e (5s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 19:12:21 (1763511141) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 19:12:47 (1763511167) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 19:13:13 (1763511193) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 19:13:21 (1763511201) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 19:13:27 (1763511207) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.58298 s, 16.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11403 s, 23.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.16604 s, 12.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51147 s, 17.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07729 s, 12.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.85124 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48593 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.957972 s, 27.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.216 s, 21.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25725 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3551 s, 11.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77438 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06636 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38914 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.669 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3755 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54387 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.77613 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.0658 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (33s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 19:14:00 (1763511240) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Tue Nov 18 19:14:39 EST 2025 All operations completed A-OK! + fsx 2 done: rc=0 Tue Nov 18 19:14:39 EST 2025 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Tue Nov 18 19:14:39 EST 2025 + fsx 4 done: rc=0 Tue Nov 18 19:14:39 EST 2025 All operations completed A-OK! + fsx 5 done: rc=0 Tue Nov 18 19:14:39 EST 2025 PASS 16k (46s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 19:14:46 (1763511286) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 19:14:56 (1763511296) mmap test1: basic mmap operation (PASS, 0.15444s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10222s) mmap test3: concurrent mmap ops on two nodes (PASS, 21.412s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3959s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16212s) mmap test6: check mmap write/read content on two nodes (PASS, 0.26808s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.94802s) mmap test11: multi-thread shared mmap (PASS, 0.9261s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0841s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 19:15:33 (1763511333) oleg138-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (3s) == sanityn test 20: test extra readahead page left in cache ============================================================== 19:15:36 (1763511336) PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 19:15:41 (1763511341) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 19:15:47 (1763511347) now is 1763511349 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 new atime is 1763511410 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 19:16:55 (1763511415) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 19:17:02 (1763511422) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 19:17:08 (1763511428) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 19:17:15 (1763511435) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 19:17:18 (1763511438) utime: good mknod times 1763511438 <= 1763511439 <= 1763511439 for /mnt/lustre/f26a utime: good mknod times 1763511438 <= 1763511439 <= 1763511439 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 19:17:24 (1763511444) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 19:17:32 (1763511452) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-11-18 19:17:33.000000000 -0500 Birth: 2025-11-18 19:17:33.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-11-18 19:17:33.000000000 -0500 Birth: 2025-11-18 19:17:33.000000000 -0500 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 19:17:42 (1763511462) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 19:17:50 (1763511470) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 19:17:58 (1763511478) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.138@tcp:/lustre 7542784 40960 7495680 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 19:18:19 (1763511499) -rw-r--r-- 1 root root 0 Nov 18 19:18 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 19:18:28 (1763511508) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42933 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 19:18:35 (1763511515) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 19:18:44 (1763511524) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 19:18:49 (1763511529) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 19:18:53 (1763511533) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 19:18:57 (1763511537) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 19:19:01 (1763511541) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid in FULL state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid in FULL state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af785e88800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9af785e88800.ost_server_uuid in FULL state after 0 sec PASS 34 (125s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 19:21:06 (1763511666) fail_loc=0x80000317 Race attempt 0 Wait for 48984 49028 for 60 sec... - open/close 702 (time 1763511683.33 total 10.01 last 70.16) - open/close 1441 (time 1763511693.33 total 20.01 last 73.89) - open/close 2307 (time 1763511703.34 total 30.01 last 86.52) - open/close 2985 (time 1763511713.34 total 40.02 last 67.79) - open/close 3660 (time 1763511723.34 total 50.02 last 67.49) total: 4000 open/close in 54.76 seconds: 73.04 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.138@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.201.138@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 19:22:17 (1763511737) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.28768 s, 40.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12697 s, 46.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.958796 s, 54.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.882759 s, 59.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7489536) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.956683 s, 54.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.972373 s, 53.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10451 s, 47.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56884 s, 33.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.78133 s, 67.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.995405 s, 52.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.06366 s, 49.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7492608) after_dd(7438336) after(7496704) PASS 36 (293s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 19:27:10 (1763512030) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 - create 1297 (time 1763512044.13 total 10.00 last 129.67) - create 2578 (time 1763512054.14 total 20.00 last 128.08) - create 3965 (time 1763512064.14 total 30.01 last 138.63) - create 5508 (time 1763512074.15 total 40.02 last 154.19) - create 6952 (time 1763512084.15 total 50.02 last 144.38) - create 8377 (time 1763512094.15 total 60.02 last 142.42) - create 9795 (time 1763512104.15 total 70.02 last 141.79) total: 10000 create in 71.26 seconds: 140.33 ops/second PASS 37 (81s) == sanityn test 39a: file mtime does not change after rename ========================================================== 19:28:31 (1763512111) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 19:28:38 (1763512118) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 19:28:45 (1763512125) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 19:28:53 (1763512133) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 19:29:02 (1763512142) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 19:29:20 (1763512160) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (14s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 19:29:34 (1763512174) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 19:29:52 (1763512192) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 19:30:10 (1763512210) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 19:30:22 (1763512222) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 19:30:35 (1763512235) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (12s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 19:30:47 (1763512247) Conflict PASS 41d (11s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 19:30:58 (1763512258) Conflict PASS 41e (10s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 19:31:08 (1763512268) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 19:31:21 (1763512281) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 19:31:34 (1763512294) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 19:31:50 (1763512310) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1012s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 19:48:42 (1763513322) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 19:48:56 (1763513336) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 19:49:08 (1763513348) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 19:49:22 (1763513362) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 19:49:35 (1763513375) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 19:49:49 (1763513389) Conflict PASS 42f (11s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 19:50:00 (1763513400) Conflict PASS 42g (11s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 19:50:11 (1763513411) Conflict PASS 42h (11s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 19:50:22 (1763513422) PASS 43a (66s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 19:51:28 (1763513488) Conflict PASS 43b (11s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 19:51:39 (1763513499) Conflict PASS 43c (11s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 19:51:50 (1763513510) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (10s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 19:52:00 (1763513520) Conflict PASS 43e (11s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 19:52:12 (1763513532) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 19:52:23 (1763513543) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (10s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 19:52:34 (1763513554) Conflict PASS 43h (12s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 19:52:45 (1763513565) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 19:52:47 (1763513567) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (99s) == sanityn test 43k: unlink vs create ==================== 19:54:26 (1763513666) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1073s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 20:12:19 (1763514739) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Nov 18 20:12:28 EST 2025 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 20:12:33 (1763514753) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 20:12:45 (1763514765) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (10s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 20:12:55 (1763514775) Conflict PASS 44d (10s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 20:13:05 (1763514785) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 20:13:17 (1763514797) Conflict PASS 44f (11s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 20:13:28 (1763514808) Conflict PASS 44g (10s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 20:13:39 (1763514819) Conflict PASS 44h (12s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 20:13:50 (1763514830) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 20:13:52 (1763514832) PASS 45a (99s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 20:15:31 (1763514931) Conflict PASS 45b (8s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 20:15:39 (1763514939) Conflict PASS 45c (9s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 20:15:48 (1763514948) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (8s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 20:15:56 (1763514956) Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 20:16:04 (1763514964) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 20:16:11 (1763514971) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (8s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 20:16:19 (1763514979) Conflict PASS 45h (8s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 20:16:27 (1763514987) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 20:16:29 (1763514989) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (718s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 20:28:27 (1763515707) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 20:28:35 (1763515715) open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 20:28:43 (1763515723) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 20:28:51 (1763515731) Conflict PASS 46d (8s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 20:28:59 (1763515739) Conflict PASS 46e (8s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 20:29:07 (1763515747) Conflict PASS 46f (8s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 20:29:16 (1763515756) Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 20:29:23 (1763515763) Conflict PASS 46h (9s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 20:29:32 (1763515772) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 20:29:33 (1763515773) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 20:29:35 (1763515775) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 20:29:36 (1763515776) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 20:29:38 (1763515778) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 20:29:39 (1763515779) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 20:29:41 (1763515781) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 20:29:42 (1763515782) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 20:29:44 (1763515784) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0128921 s, 794 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 20:29:52 (1763515792) 0+1 records in 0+1 records out 158 bytes copied, 0.00757346 s, 20.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216302) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 20:29:57 (1763515797) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00489196 s, 209 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292272 s, 35.9 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 20:30:12 (1763515812) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0132654 s, 77.2 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 20:30:20 (1763515820) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199719 s, 52.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 20:30:24 (1763515824) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2449 0x991 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2449 0x991 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 20:30:29 (1763515829) llite.lustre-ffff9af785e88800.xattr_cache=1 llite.lustre-ffff9af798626000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:29.000000000 -0500 Modify: 2025-11-18 20:30:29.000000000 -0500 Change: 2025-11-18 20:30:29.000000000 -0500 Birth: 2025-11-18 20:30:29.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:35.000000000 -0500 Modify: 2025-11-18 20:30:35.000000000 -0500 Change: 2025-11-18 20:30:35.000000000 -0500 Birth: 2025-11-18 20:30:35.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:40.000000000 -0500 Modify: 2025-11-18 20:30:40.000000000 -0500 Change: 2025-11-18 20:30:40.000000000 -0500 Birth: 2025-11-18 20:30:40.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:40.000000000 -0500 Modify: 2025-11-18 20:30:41.000000000 -0500 Change: 2025-11-18 20:30:41.000000000 -0500 Birth: 2025-11-18 20:30:40.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:46.000000000 -0500 Modify: 2025-11-18 20:30:47.000000000 -0500 Change: 2025-11-18 20:30:47.000000000 -0500 Birth: 2025-11-18 20:30:46.000000000 -0500 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:30:46.000000000 -0500 Modify: 2025-11-18 20:30:47.000000000 -0500 Change: 2025-11-18 20:30:47.000000000 -0500 Birth: 2025-11-18 20:30:46.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 20:30:54 (1763515854) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 20:31:03 (1763515863) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 20:31:12 (1763515872) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 20:31:25 (1763515885) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 20:31:36 (1763515896) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 20:31:37 (1763515897) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 20:31:51 (1763515911) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 20:32:07 (1763515927) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1763515927 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.19684 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (53s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 20:33:01 (1763515981) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1763515982 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.71276 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (51s) == sanityn test 60: Verify data_version behaviour ======== 20:33:51 (1763516031) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 20:33:54 (1763516034) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:33:58 (1763516038) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:34:02 (1763516042) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00473776 s, 13.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:34:04 (1763516044) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00683946 s, 9.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:34:06 (1763516046) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:34:07 (1763516047) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00813053 s, 8.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:34:09 (1763516049) llite.lustre-ffff9af785e88800.xattr_cache=1 llite.lustre-ffff9af798626000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:34:12 (1763516052) llite.lustre-ffff9af785e88800.xattr_cache=1 llite.lustre-ffff9af798626000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:34:15 (1763516055) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00382982 s, 267 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00589583 s, 174 kB/s 229083: get lock2 229086: get lock1 229086: done 229086: get lock2 229083: done 229083: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 229083: cannot get lock1: Resource deadlock avoided 229086: done 229086: exit rc=0 229083: done 229083: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:34:22 (1763516062) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0293033 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242997 s, 43.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229667 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 20:34:31 (1763516071) opening files: ........ closing files: ........ PASS 76 (30s) == sanityn test 77a: check FIFO NRS policy =============== 20:35:00 (1763516100) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190402 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344724 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456531 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102794 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101099 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118836 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117765 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991999 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118176 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128756 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16173 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14111 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173372 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173036 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182103 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182755 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180846 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616897 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589918 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346319 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545362 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386377 s, 27.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575599 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481843 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613931 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487869 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389704 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559312 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647922 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553028 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478415 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563472 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767063 s, 13.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (5s) == sanityn test 77b: check CRR-N NRS policy ============== 20:35:05 (1763516105) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241065 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305357 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304188 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105508 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121968 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120108 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967447 s, 10.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130675 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131119 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122882 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114135 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12479 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123838 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136284 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128051 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132178 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157147 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829623 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010258 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181054 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173397 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220744 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298177 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222202 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856063 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121188 s, 86.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166347 s, 63.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150337 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00661069 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106351 s, 98.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114792 s, 91.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816534 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00732364 s, 143 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172962 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370844 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396561 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833476 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111759 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133022 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120724 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132986 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109088 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134283 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140585 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125266 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131175 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145337 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124398 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150607 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148268 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312445 s, 336 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390426 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490848 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271082 s, 387 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451094 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346473 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306062 s, 343 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578036 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030236 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272031 s, 38.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251361 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231089 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199266 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176722 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147313 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128921 s, 81.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (6s) == sanityn test 77c: check ORR NRS policy ================ 20:35:11 (1763516111) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253854 s, 66.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419546 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029425 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405222 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920546 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124094 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130954 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123639 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132032 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116907 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125714 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123613 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141453 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136215 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153343 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142453 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139631 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191593 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034961 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407767 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162182 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383723 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234073 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253874 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557377 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251642 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263405 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275341 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424456 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285675 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500606 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526628 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509252 s, 20.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.204773 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056149 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591302 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114118 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129606 s, 8.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119194 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14408 s, 7.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140114 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135991 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101471 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145604 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134118 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132497 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149455 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154953 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185505 s, 5.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153365 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306797 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239398 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305447 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158971 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244974 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216661 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270046 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270306 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463045 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284336 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409145 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365047 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303628 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349896 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00930519 s, 113 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.205855 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286528 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327615 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394323 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508571 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483093 s, 21.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998431 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114905 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115224 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964878 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118509 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124084 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12979 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127035 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130319 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153601 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150234 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258345 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184864 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239822 s, 43.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314451 s, 33.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024605 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421648 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264174 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293659 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029749 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277612 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306146 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313836 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325725 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348096 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340677 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383784 s, 27.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (7s) == sanityn test 77d: check TRR nrs policy ================ 20:35:18 (1763516118) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222243 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269882 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357158 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085529 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811488 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904932 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110742 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895371 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108518 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125842 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120053 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132461 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104585 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108963 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132101 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115726 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150829 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186295 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00296621 s, 354 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642752 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477305 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619271 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583657 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373109 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270403 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270704 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223302 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00599131 s, 175 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216746 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181641 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221356 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129867 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169526 s, 61.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.188696 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376594 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415486 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738295 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794876 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742922 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816091 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108592 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109003 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108733 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955229 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110324 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106609 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116361 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14677 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128264 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190601 s, 55.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229615 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208978 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281319 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290183 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373768 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285722 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421229 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035024 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394598 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00590529 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222931 s, 47.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162318 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282823 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293858 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397142 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0075288 s, 139 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168046 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286219 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374643 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109443 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999279 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101239 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116519 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113068 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115146 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128529 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116252 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132962 s, 7.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125427 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125348 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113864 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130151 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145799 s, 7.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213468 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183723 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274959 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380568 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252183 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274689 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034984 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00274682 s, 382 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311098 s, 33.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03347 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134553 s, 77.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113307 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103316 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00772438 s, 136 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189798 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063003 s, 166 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (8s) == sanityn test 77e: check TBF NID nrs policy ============ 20:35:26 (1763516126) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.38@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.272146 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319045 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447385 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927204 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680053 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118977 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105565 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111144 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989203 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135496 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127683 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139087 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142804 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125291 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156665 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141902 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123762 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988236 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105502 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112582 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124942 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112954 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11619 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118231 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11585 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11983 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116971 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120121 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118442 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125763 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120271 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127014 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122746 s, 8.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201219 s, 83.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363207 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361502 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599508 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929334 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671368 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105663 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124892 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130599 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125186 s, 8.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115062 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11311 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118676 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133017 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117709 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126481 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119612 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17005 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175961 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172626 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176602 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17127 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181288 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175093 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182976 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176641 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17893 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185605 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178893 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193021 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182391 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185054 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185203 s, 5.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183545 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436001 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407331 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048294 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461034 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441975 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992737 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853866 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114868 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105789 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103504 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122366 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115625 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115722 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14048 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141735 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143597 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012574 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263515 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179457 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196346 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228929 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183106 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216933 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306738 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264047 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285895 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298202 s, 35.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294857 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358303 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276775 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337237 s, 31.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319248 s, 32.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179976 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281645 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384285 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398844 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884354 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784734 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834613 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127238 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109268 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133087 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142858 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140474 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114139 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140078 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131066 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123872 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144177 s, 7.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765791 s, 137 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166409 s, 63.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399659 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145271 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168985 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327716 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175831 s, 59.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00882389 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452229 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044433 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305418 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472968 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130045 s, 80.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522681 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334506 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249413 s, 42.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:35:36 (1763516136) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195773 s, 85.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326799 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401825 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286588 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113403 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110738 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125425 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113321 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125789 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125315 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121232 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13193 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151237 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147794 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130357 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131371 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146777 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195865 s, 53.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166934 s, 62.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315527 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016616 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185576 s, 56.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183542 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195183 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205902 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348595 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212879 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459441 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227412 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398252 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272154 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271004 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572715 s, 183 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.213199 s, 78.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389616 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241621 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374845 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993279 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097993 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112387 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121327 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129374 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131387 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139469 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133955 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140412 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129659 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110298 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121076 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137832 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021961 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287399 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217162 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246855 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219576 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026035 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024488 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029806 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287479 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274109 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242769 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268201 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296539 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290795 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029274 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343289 s, 30.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1716 s, 97.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571112 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635109 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705769 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633236 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719676 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690969 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661663 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773695 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884446 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745825 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887127 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121262 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117761 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130786 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129009 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124959 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00932053 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268543 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404029 s, 260 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101876 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252899 s, 41.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404591 s, 259 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418092 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896329 s, 117 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689325 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371646 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089649 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345043 s, 30.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323952 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466191 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00261705 s, 401 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127606 s, 82.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17223 s, 97.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490884 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535446 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580584 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490075 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553629 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792924 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939002 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900171 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10829 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108399 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134052 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118598 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128523 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14117 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155143 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af798626000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9af798626000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124967 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993518 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121307 s, 86.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187705 s, 55.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162871 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164821 s, 63.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295378 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222464 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230546 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232741 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235021 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024961 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249802 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264251 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274659 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242187 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317229 s, 33.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (11s) == sanityn test 77g: Change TBF type directly ============ 20:35:47 (1763516147) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (5s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:35:52 (1763516152) oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg138-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg138-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg138-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg138-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 20:35:57 (1763516157) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:36:06 (1763516166) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155471 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208894 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025349 s, 41.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678578 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588211 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089341 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813194 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068274 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688848 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102665 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953222 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085773 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107841 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968237 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108954 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883766 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917718 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237683 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230258 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020248 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247475 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231891 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251258 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284714 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237321 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315064 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270887 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249213 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291895 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321424 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303436 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276915 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324491 s, 32.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99754 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4707 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (52s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:36:58 (1763516218) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23572 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031215 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639229 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768433 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702898 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460858 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417544 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095892 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689156 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148878 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146686 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166222 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170073 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18343 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179863 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195271 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183905 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227225 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222057 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025137 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259025 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317091 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311966 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031382 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287982 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292991 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282819 s, 37.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316814 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343773 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397005 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408992 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031602 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371846 s, 28.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8198 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9631 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477826 s, 35.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397622 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454549 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459972 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596927 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515478 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732839 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829143 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677393 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977826 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096149 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131272 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105056 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129453 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108903 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123078 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151877 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193268 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217608 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218957 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235617 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213435 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240351 s, 43.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233326 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230205 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204256 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254583 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252738 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025367 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252293 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240514 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280922 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295397 s, 35.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6905 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.952 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (135s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:39:13 (1763516353) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22945 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725137 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067776 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944453 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810314 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846077 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767183 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097791 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106629 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150765 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150688 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154559 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16053 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165682 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169897 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156074 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150988 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217157 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224722 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230839 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186949 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238068 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184052 s, 57.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172356 s, 60.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214751 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021289 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020357 s, 51.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271622 s, 38.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237218 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243535 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251808 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397923 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578528 s, 181 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7511 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9522 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23382 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023074 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479015 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411786 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587565 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597881 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372014 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04404 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546706 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145833 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145781 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16432 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175863 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183643 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182229 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181184 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203623 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201164 s, 52.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234749 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150771 s, 69.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134814 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275511 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201381 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269711 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295252 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212635 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278771 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254287 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222313 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272367 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271625 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276286 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315245 s, 33.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7425 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9466 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (135s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:41:29 (1763516489) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.277189 s, 60.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297178 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411837 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936666 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813122 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789032 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880439 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705959 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840851 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986603 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100934 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859756 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669407 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104839 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125411 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124278 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140848 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176567 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277458 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194556 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216385 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214549 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260685 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232076 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399101 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251229 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346802 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258155 s, 40.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286119 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289238 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294683 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302079 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305368 s, 34.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14042 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77514 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.38@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.38@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.207684 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197304 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357058 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029904 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731481 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588922 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729599 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979746 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105813 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107209 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115335 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131525 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132306 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175339 s, 6.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160376 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199907 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2194 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188817 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157638 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030764 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017886 s, 58.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122705 s, 85.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00750361 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228884 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398553 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434821 s, 241 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00573375 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314513 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424426 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435097 s, 241 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329115 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274595 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358773 s, 29.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1383 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8273 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.38@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.34346 s, 48.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283892 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340469 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800039 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110116 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108894 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115434 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115425 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121546 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104854 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120123 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11743 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115768 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107135 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126563 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138038 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120751 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435433 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407616 s, 257 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390127 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465851 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520492 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566476 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321476 s, 326 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353209 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368486 s, 28.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340568 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307324 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259679 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167055 s, 62.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122505 s, 85.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102345 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237337 s, 44.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00944 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94581 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174416 s, 96.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270766 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254168 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925422 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980524 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964179 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904445 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910569 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103197 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109619 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923039 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102167 s, 10.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10273 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103725 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924352 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934658 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990693 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417479 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016529 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00788921 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435073 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398376 s, 263 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00657551 s, 159 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039924 s, 263 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668222 s, 157 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436023 s, 240 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535898 s, 196 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283384 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316044 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042726 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483092 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731412 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033326 s, 315 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84656 s, 10.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.782 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95177 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77872 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175036 s, 95.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317739 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482277 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439373 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639217 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645101 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059037 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602638 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852194 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073627 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101709 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962384 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895146 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996812 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109728 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114476 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896949 s, 11.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116935 s, 89.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252296 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171837 s, 61.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016866 s, 62.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182869 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287852 s, 36.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269177 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269844 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197981 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200547 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224161 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239031 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221997 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232418 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257827 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245197 s, 42.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5279 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4709 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241065 s, 69.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323669 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281452 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069045 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619451 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658899 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833311 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921916 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866319 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846208 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949562 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867979 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104583 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101342 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134031 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122531 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140824 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005543 s, 189 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780045 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135361 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211182 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110435 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349724 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126882 s, 82.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041577 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257488 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370716 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267925 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317722 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434701 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336705 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682429 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457977 s, 229 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02464 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76738 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186798 s, 89.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435848 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402454 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105842 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13408 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107136 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107555 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10349 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131108 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122858 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14238 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136812 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134384 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154235 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177103 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210525 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21774 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165974 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119763 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270656 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203414 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240642 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234187 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246908 s, 42.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248574 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257957 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027793 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318248 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306902 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305275 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315137 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278591 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344231 s, 30.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84376 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74918 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79971 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76944 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (362s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:47:31 (1763516851) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.294823 s, 56.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312574 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289628 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791363 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109152 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117243 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106939 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924092 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105259 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113616 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105687 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130087 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133928 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121808 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130041 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134409 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139464 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129857 s, 80.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547001 s, 192 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00386278 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197406 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027533 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103318 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141618 s, 74.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122659 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276501 s, 379 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161445 s, 64.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336819 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411248 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712875 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407699 s, 257 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432171 s, 243 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434559 s, 241 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09477 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77528 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (39s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:48:10 (1763516890) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.38@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.38@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20655 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253944 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235197 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117424 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114682 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103238 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973908 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118949 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127993 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121176 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126273 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125392 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119261 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129753 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129392 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180128 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196793 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258645 s, 405 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516158 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363261 s, 289 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108254 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884901 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378614 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031251 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273552 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390845 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382572 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515387 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488209 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149602 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107358 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0077013 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910625 s, 115 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13936 s, 20.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77476 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (39s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:48:49 (1763516929) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.38@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164452 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252705 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266953 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329464 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432853 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517712 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649806 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345978 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325377 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636027 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642462 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970133 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10612 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605469 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956503 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120348 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105659 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416688 s, 252 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019932 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190417 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197744 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124788 s, 84.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175666 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251801 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199181 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160356 s, 65.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200855 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004847 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232182 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221751 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221151 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283822 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258697 s, 405 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95234 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9776 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:49:21 (1763516961) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135029 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210612 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388735 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539415 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524999 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615644 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789184 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640732 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781466 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722729 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962476 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851819 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984179 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225124 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213321 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259195 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275296 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167483 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207079 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163362 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159005 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776058 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161103 s, 65.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226399 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240273 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022159 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206878 s, 50.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271441 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372297 s, 282 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270547 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278211 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284117 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552477 s, 190 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77656 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72951 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96125 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74152 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:50:39 (1763517039) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163929 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354576 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675342 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881336 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585508 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988052 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763573 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109794 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924336 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114189 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099025 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112473 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842626 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130679 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115233 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126672 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118551 s, 8.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411696 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146786 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018021 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195188 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222832 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427336 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197088 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235436 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228987 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215841 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211924 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300885 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237575 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233976 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243952 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237336 s, 442 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6859 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9629 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (63s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:51:42 (1763517102) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131976 s, 127 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280265 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505123 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449123 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417307 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607401 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570412 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610782 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726366 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513699 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827558 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906229 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985603 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108897 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750353 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103923 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103766 s, 10.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009428 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149037 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159887 s, 65.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159741 s, 65.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207635 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081586 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167454 s, 62.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148707 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193206 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665383 s, 158 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208622 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00322275 s, 325 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204674 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218843 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361289 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500952 s, 209 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93631 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73605 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg138-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131495 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292115 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585976 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695221 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930754 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511303 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959506 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851816 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822074 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979774 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850313 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864583 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802809 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980475 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939498 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166981 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159761 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155032 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196524 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208092 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177373 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026511 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181956 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230625 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405856 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027243 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222561 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194155 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310156 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020567 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251953 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265673 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273595 s, 38.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79987 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73402 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77113 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73351 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (115s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:53:37 (1763517217) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre opts: Stopping clients: oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg138-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg138-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.38,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg138-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.38,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg138-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.38,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/57417/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.38,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg138-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Started clients oleg138-client.virtnet: 192.168.201.138@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Started clients oleg138-client.virtnet: 192.168.201.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9af784e86000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af791182000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af784e86000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af791182000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237299 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227801 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297164 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874497 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876771 s, 12.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825065 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873897 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776612 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936269 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802621 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887884 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949767 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922462 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263249 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263235 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302059 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288266 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151557 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157068 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212308 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255876 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211389 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175821 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144356 s, 72.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233575 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110624 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00634525 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228928 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222786 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246457 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275823 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238641 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268605 s, 39.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97488 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96914 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253207 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397406 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496443 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066581 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409334 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786819 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719397 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981216 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930883 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804432 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108516 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136542 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126843 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230335 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242322 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277173 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276512 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136361 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155939 s, 67.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020653 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230149 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215542 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159187 s, 65.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202743 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844317 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343756 s, 305 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194399 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236028 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206461 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220731 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235284 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245434 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022122 s, 47.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01792 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73478 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (117s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:55:34 (1763517334) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg138-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:55:42 (1763517342) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:55:45 (1763517345) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg138-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.74716 s, 280 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg138-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0173 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:56:36 (1763517396) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121189 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469847 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544588 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567601 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409674 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483644 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690177 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750613 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597388 s, 17.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404088 s, 25.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9af791182000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100246 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103109 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117211 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967569 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105714 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9af791182000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117153 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121493 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119835 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185627 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171379 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206062 s, 50.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137904 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148252 s, 70.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186791 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233115 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225208 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201162 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184186 s, 56.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00509786 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259349 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233098 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011758 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856334 s, 122 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg138-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg138-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98553 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96917 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.243936 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393229 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870228 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939229 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910587 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10423 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938321 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118318 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947153 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838158 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102466 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954461 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974995 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257771 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264911 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29949 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307556 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225094 s, 46.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238082 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023704 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238845 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242414 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320376 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259602 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259645 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237183 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340599 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278228 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473686 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390987 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321461 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029685 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324888 s, 32.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg138-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg138-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92184 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97143 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (67s) == sanityn test 77o: Changing rank should not panic ====== 20:57:43 (1763517463) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:57:47 (1763517467) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (38s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:58:25 (1763517505) mds.MDS.mdt.nrs_policies=tbf oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg138-client: oleg138-server: ssh exited with exit code 36 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg138-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg138-client: oleg138-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:58:37 (1763517517) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269588 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:59:19 (1763517559) oleg138-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg138-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg138-client: oleg138-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 20:59:22 (1763517562) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935153 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 20:59:22.000000000 -0500 Modify: 2025-11-18 20:59:22.000000000 -0500 Change: 2025-11-18 20:59:22.000000000 -0500 Birth: 2025-11-18 20:59:22.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:59:35 (1763517575) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:59:36 (1763517576) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:59:37 (1763517577) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:59:38 (1763517578) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:59:39 (1763517579) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:59:40 (1763517580) total: 2500 open/close in 5.58 seconds: 448.28 ops/second snapshot_time 1763517603.597422960 secs.nsecs start_time 1763517581.213119395 secs.nsecs elapsed_time 22.384303565 secs.nsecs open 2505 samples [usecs] 92 3633 800215 306160747 close 2504 samples [usecs] 68 2371 263618 38454246 mknod 2501 samples [usecs] 179 3526 696414 239479176 unlink 2500 samples [usecs] 285 65227 1558524 9130780954 mkdir 2 samples [usecs] 614 809 1423 1031477 rmdir 2 samples [usecs] 1997 3505 5502 16273034 rename 2500 samples [usecs] 259 14762 3383787 6837054615 getattr 3760 samples [usecs] 7 1704 80118 7339994 parallel_rename_file 2500 samples [usecs] 259 14762 3383787 6837054615 crossdir_rename 2500 samples [usecs] 259 14762 3383787 6837054615 PASS 81d (25s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 21:00:05 (1763517605) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7595 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7595 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 21:00:07 (1763517607) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 21:00:08 (1763517608) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 21:00:16 (1763517616) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1763517616 Finishing test test1 at 1763517617 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 21:00:18 (1763517618) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 21:00:19 (1763517619) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 21:00:21 (1763517621) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 21:00:22 (1763517622) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5818 0x16ba 0x240000400 1 5784 0x1698 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5819 0x16bb 0x240000400 1 5785 0x1699 0x280000400 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 21:00:30 (1763517630) osc.lustre-OST0000-osc-ffff9af784e86000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9af791182000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9af784e86000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9af791182000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0272353 s, 301 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.095 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 287552 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.012426 s, 330 kB/s osc.lustre-OST0000-osc-ffff9af784e86000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9af791182000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af784e86000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9af791182000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 21:00:40 (1763517640) llite.lustre-ffff9af784e86000.fast_read=0 llite.lustre-ffff9af791182000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0157051 s, 134 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9af784e86000.fast_read=1 llite.lustre-ffff9af791182000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 21:00:55 (1763517655) llite.lustre-ffff9af784e86000.fast_read=0 llite.lustre-ffff9af791182000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00153048 s, 8.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9af784e86000.fast_read=1 llite.lustre-ffff9af791182000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 21:01:03 (1763517663) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 21:01:04 (1763517664) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00172727 s, 2.4 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 21:01:07 (1763517667) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0484866 s, 43.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 21:01:09 (1763517669) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0580555 s, 36.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 21:01:11 (1763517671) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00150054 s, 8.0 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 21:01:13 (1763517673) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000650812 s, 6.3 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 21:01:15 (1763517675) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000590695 s, 6.9 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 21:01:18 (1763517678) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000743477 s, 5.5 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 21:01:22 (1763517682) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1763517683 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1763517685 Finishing test test23 at 1763517685 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 21:01:29 (1763517689) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 21:01:31 (1763517691) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 21:01:48 (1763517708) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 21:01:49 (1763517709) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00306793 s, 334 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937688 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 21:01:50.000000000 -0500 Modify: 2025-11-18 21:01:50.000000000 -0500 Change: 2025-11-18 21:01:50.000000000 -0500 Birth: 2025-11-18 21:01:50.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937688 -rw-r--r-- 1 0 0 1763517710 1763517710 1763517710 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937688 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 21:01:50.000000000 -0500 Modify: 2025-11-18 21:01:50.000000000 -0500 Change: 2025-11-18 21:01:50.000000000 -0500 Birth: 2025-11-18 21:01:50.000000000 -0500 1763517710 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 21:01:52 (1763517712) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 21:01:54 (1763517714) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0859602 s, 122 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5791 0x169f 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:01:58 (1763517718) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0990047 s, 106 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5828 0x16c4 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7595 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 21:02:07 (1763517727) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00278965 s, 3.6 MB/s llite.lustre-ffff9af784e86000.hybrid_io=0 llite.lustre-ffff9af791182000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.38228 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 21:02:14 (1763517734) 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre 192.168.201.138@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 PASS 109 (543s) == sanityn test 110: do not grant another lock on resend ========================================================== 21:11:17 (1763518277) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 21:11:18.000000000 -0500 Modify: 2025-11-18 21:11:18.000000000 -0500 Change: 2025-11-18 21:11:18.000000000 -0500 Birth: 2025-11-18 21:11:17.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg138-client.virtnet oleg138-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 21:12:22 (1763518342) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 21:12:23 (1763518343) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 21:12:24 (1763518344) lustre-OST0000-osc-ffff9af789345000 active. lustre-OST0001-osc-ffff9af789345000 active. lustre-OST0000-osc-ffff9af789345000 active. lustre-OST0001-osc-ffff9af789345000 active. MGC192.168.201.138@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 21:12:27 (1763518347) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 21:12:28 (1763518348) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 21:12:29 (1763518349) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 21:12:30 (1763518350) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0923 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.093 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0969 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.113 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1024 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0996 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0954 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.145 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1503 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1642 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.168 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1788 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1607 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1577 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2407 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2381 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2535 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2646 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af789345000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9af789345000.ost_server_uuid in FULL state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 7767 sec ============== 21:13:39 (1763518419) === sanityn: start cleanup 21:13:39 (1763518419) === Stopping clients: oleg138-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg138-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 21:14:29 (1763518469) ===