-----============= acceptance-small: sanityn ============----- Mon Dec 22 09:12:54 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg228-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 09:13:14 (1766412794) === Starting client oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Mount client oleg228-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Started clients oleg228-client.virtnet: 192.168.202.128@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg228-client.virtnet: executing check_config_client /mnt/lustre oleg228-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg228-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c1106845000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1119f79000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1106845000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1119f79000.idle_timeout=debug disable quota as required oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 09:13:43 (1766412823) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174755 s, 60.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7542] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 09:13:45 (1766412825) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 09:13:52 (1766412832) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 09:13:58 (1766412838) -rw-r--r-- 1 root root 0 Dec 22 09:14 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 09:14:04 (1766412844) -rw-r--r-- 1 root root 0 Dec 22 09:14 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 09:14:10 (1766412850) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 09:14:16 (1766412856) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 09:14:21 (1766412861) total 1 -rwxrwxrwx 1 root root 0 Dec 22 09:14 f2a -rwxrwxrwx 1 root root 0 Dec 22 09:14 f2b total 1 -rwxrwxrwx 1 root root 0 Dec 22 09:14 f2a -rwxrwxrwx 1 root root 0 Dec 22 09:14 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0206286 s, 24.8 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 09:14:27 (1766412867) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 09:14:30 (1766412870) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281364 s, 7.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 09:14:36 (1766412876) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 09:14:41 (1766412881) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 09:14:48 (1766412888) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 09:14:54 (1766412894) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 09:15:00 (1766412900) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 09:15:05 (1766412905) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 09:15:11 (1766412911) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 09:15:17 (1766412917) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525632 s, 19.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0703203 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 09:15:23 (1766412923) 1+0 records in 1+0 records out 1 byte copied, 0.00468199 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116764 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00917373 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101462 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0157391 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0420291 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00745872 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100945 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121054 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112235 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123256 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0241493 s, 0.0 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 09:15:32 (1766412932) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0701161 s, 43.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.040043 s, 102 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000217079 s, 14.2 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 09:15:38 (1766412938) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 09:15:44 (1766412944) warning: '-runtime' deprecated, use '-t runtime' instead - link 444 (time 1766412956.76 total 10.02 last 44.32) - link 853 (time 1766412966.78 total 20.04 last 40.83) - link 1246 (time 1766412976.79 total 30.05 last 39.26) - link 1654 (time 1766412986.80 total 40.05 last 40.77) - link 2042 (time 1766412996.80 total 50.06 last 38.77) - link 2487 (time 1766413006.81 total 60.07 last 44.47) - link 2929 (time 1766413016.84 total 70.10 last 44.06) - link 3379 (time 1766413026.85 total 80.11 last 44.97) - link 3831 (time 1766413036.85 total 90.11 last 45.20) total: 4281 link in 100.03 seconds: 42.80 ops/second using seed 1944681674 running for 100 seconds - stat 9078 (time 1766412958 ; total 11 ; last 11) - stat 10000 (time 1766412958 ; total 11 ; last 0) - stat 17592 (time 1766412969 ; total 22 ; last 11) - stat 20000 (time 1766412972 ; total 25 ; last 3) - stat 28837 (time 1766412983 ; total 36 ; last 11) - stat 30000 (time 1766412984 ; total 37 ; last 1) - stat 39185 (time 1766412995 ; total 48 ; last 11) - stat 40000 (time 1766412995 ; total 48 ; last 0) - stat 45186 (time 1766413006 ; total 59 ; last 11) - stat 50000 (time 1766413016 ; total 69 ; last 10) - stat 54112 (time 1766413027 ; total 80 ; last 11) - stat 58857 (time 1766413038 ; total 91 ; last 11) - stat 60000 (time 1766413040 ; total 93 ; last 2) total: 62703 stats in 100 seconds: 627.030029 stats/second - unlinked 0 (time 1766412997 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile4281) error: No such file or directory total: 4280 unlinks in 93 seconds: 46.021507 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21340) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21342) - No such process PASS 12 (151s) == sanityn test 13: test directory page revocation ======= 09:18:16 (1766413096) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:18:24 (1766413104) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 22527 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:18:30 (1766413110) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 23106 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:18:36 (1766413116) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23684 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:18:42 (1766413122) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24267 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:18:49 (1766413129) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24849 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:18:56 (1766413136) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (45s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:19:41 (1766413181) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.447938 s, 18.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.391395 s, 21.4 MB/s Chance of close/open is 1 in 50 Seed set to 3184 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1766413185.652796 write 0x097f33 thru 0x0a00e9 (0x81b7 bytes) 000004[1] 1766413185.695948 mapread 0x06578c thru 0x074140 (0xe9b5 bytes) 000005[0] 1766413185.766483 write 0x922c9d thru 0x92f670 (0xc9d4 bytes) 000006[0] 1766413185.814256 mapwrite 0x019767 thru 0x025721 (0xbfbb bytes) truncating to largest ever: 0x7707fb 000020[1] 1766413186.251529 trunc from 0x92f671 to 0x7707fa (0x1bee76 bytes) 000022[0] 1766413186.317063 write 0x29776e thru 0x2a0093 (0x8926 bytes) 000026[0] 1766413186.396789 write 0x866a1d thru 0x8766d4 (0xfcb8 bytes) 000032[0] 1766413186.482374 write 0x76cf80 thru 0x76e303 (0x1384 bytes) 000033[1] 1766413186.502062 write 0x92f93f thru 0x9337d7 (0x3e99 bytes) 000035[1] 1766413186.545918 mapread 0x5d3705 thru 0x5d9350 (0x5c4c bytes) 000040[1] 1766413186.590521 mapwrite 0x92b644 thru 0x937e52 (0xc80f bytes) 000041[0] 1766413186.718374 read 0x13c988 thru 0x142e00 (0x6479 bytes) 000043[0] 1766413186.753919 mapread 0x19aec6 thru 0x1a6586 (0xb6c1 bytes) truncating to largest ever: 0x7afec2 000045[0] 1766413186.802044 trunc from 0x937e53 to 0x7afec1 (0x187f91 bytes) 000046[0] 1766413186.855886 trunc from 0x7afec2 to 0x6bfab5 (0xf040c bytes) truncating to largest ever: 0x930f8a 000047[0] 1766413186.889073 trunc from 0x6bfab6 to 0x930f89 (0x2714d4 bytes) 000048[1] 1766413186.910279 mapread 0x3bf91a thru 0x3cedb6 (0xf49d bytes) 000049[1] 1766413186.949655 mapwrite 0x6bb0be thru 0x6c76d5 (0xc618 bytes) 000051[0] 1766413187.091748 mapread 0x30b913 thru 0x30bbe7 (0x2d5 bytes) 000055[1] 1766413187.159923 mapwrite 0x8b86fe thru 0x8c3227 (0xab2a bytes) 000056[1] 1766413187.283140 mapwrite 0x662d83 thru 0x6639e0 (0xc5e bytes) 000057[1] 1766413187.404386 read 0x578206 thru 0x57e7ec (0x65e7 bytes) 000058[0] 1766413187.419965 read 0x770d41 thru 0x77b200 (0xa4c0 bytes) 000059[1] 1766413187.456957 mapread 0x0e4f08 thru 0x0f49a9 (0xfaa2 bytes) 000061[0] 1766413187.481862 mapread 0x8e7236 thru 0x8e8c4a (0x1a15 bytes) 000062[0] 1766413187.511347 read 0x179feb thru 0x17fdc3 (0x5dd9 bytes) 000063[1] 1766413187.520697 write 0x4ebec4 thru 0x4fb072 (0xf1af bytes) 000065[1] 1766413187.541063 trunc from 0x930f8a to 0x7ed498 (0x143af1 bytes) 000066[0] 1766413187.574838 read 0x590751 thru 0x595cab (0x555b bytes) 000067[0] 1766413187.629276 write 0x81af09 thru 0x81f288 (0x4380 bytes) 000068[0] 1766413187.656075 write 0x52a5ea thru 0x52d6ae (0x30c5 bytes) 000071[1] 1766413187.731099 mapread 0x04fab2 thru 0x05c894 (0xcde3 bytes) 000072[1] 1766413187.803152 write 0x172e63 thru 0x1757c0 (0x295e bytes) 000075[1] 1766413187.875257 trunc from 0x81f289 to 0x6178e6 (0x2079a2 bytes) 000077[0] 1766413187.913359 write 0x95578e thru 0x95e0bb (0x892e bytes) 000079[0] 1766413188.027502 mapwrite 0x78f879 thru 0x798ad4 (0x925c bytes) 000080[1] 1766413188.252238 mapwrite 0x7ec4e5 thru 0x7fa1c2 (0xdcde bytes) 000081[1] 1766413188.743971 write 0x3de773 thru 0x3df223 (0xab1 bytes) 000082[1] 1766413188.797968 write 0x5227a4 thru 0x52bbe2 (0x943f bytes) 000084[0] 1766413188.867217 mapread 0x40364f thru 0x40fbf7 (0xc5a9 bytes) 000085[1] 1766413188.957552 mapread 0x79088f thru 0x79c379 (0xbaeb bytes) 000086[1] 1766413189.005769 trunc from 0x95e0bc to 0x5801b0 (0x3ddf0b bytes) 000087[0] 1766413189.053718 mapread 0x48ccdb thru 0x4945d6 (0x78fc bytes) 000091[1] 1766413189.169039 mapread 0x41ccd7 thru 0x4232bc (0x65e6 bytes) 000092[1] 1766413189.233073 trunc from 0x5801b1 to 0x237fc6 (0x3481ea bytes) 000094[1] 1766413189.342668 mapwrite 0x0d2982 thru 0x0e032c (0xd9ab bytes) 000095[1] 1766413189.488026 trunc from 0x237fc7 to 0x126fef (0x110fd7 bytes) 000096[0] 1766413189.517837 mapwrite 0x669558 thru 0x66d7bd (0x4266 bytes) 000099[1] 1766413189.644227 mapwrite 0x421ec1 thru 0x422e3d (0xf7d bytes) 000100[0] 1766413189.732823 read 0x53063c thru 0x532ee8 (0x28ad bytes) 000101[1] 1766413189.782190 mapread 0x16aa8d thru 0x17658e (0xbb02 bytes) 000104[1] 1766413189.861645 read 0x1d72a0 thru 0x1e0094 (0x8df5 bytes) 000105[1] 1766413189.871519 mapwrite 0x002cc4 thru 0x00d650 (0xa98d bytes) 000107[1] 1766413190.070626 mapwrite 0x936ac2 thru 0x93cc38 (0x6177 bytes) 000108[0] 1766413190.222232 write 0x63534d thru 0x636428 (0x10dc bytes) 000111[1] 1766413190.340749 write 0x4d74b2 thru 0x4dfa15 (0x8564 bytes) 000114[0] 1766413190.434155 write 0x843ec1 thru 0x84d3cb (0x950b bytes) 000115[0] 1766413190.499399 write 0x0de310 thru 0x0e6813 (0x8504 bytes) 000116[0] 1766413190.534105 mapread 0x7b323c thru 0x7c1f9f (0xed64 bytes) 000121[1] 1766413190.629011 trunc from 0x93cc39 to 0x83a781 (0x1024b7 bytes) 000125[1] 1766413190.765774 write 0x377804 thru 0x382e75 (0xb672 bytes) 000127[0] 1766413190.780704 mapread 0x3a9bf1 thru 0x3b915e (0xf56e bytes) 000128[0] 1766413190.865597 mapread 0x37c607 thru 0x380c8a (0x4684 bytes) 000131[0] 1766413190.886922 write 0x1a183e thru 0x1a2e53 (0x1616 bytes) 000132[1] 1766413190.921507 read 0x2d50c3 thru 0x2dee96 (0x9dd4 bytes) 000134[0] 1766413190.965771 write 0x791f18 thru 0x79e4db (0xc5c4 bytes) 000137[1] 1766413191.032904 write 0x0bba6d thru 0x0ca89c (0xee30 bytes) 000141[1] 1766413191.207617 trunc from 0x83a782 to 0x3df5ad (0x45b1d4 bytes) 000142[0] 1766413191.320937 trunc from 0x3df5ae to 0x15622a (0x289383 bytes) 000143[1] 1766413191.389279 write 0x147e16 thru 0x15261b (0xa806 bytes) 000145[1] 1766413191.466426 write 0x8dc411 thru 0x8e3533 (0x7123 bytes) 000148[1] 1766413191.531222 mapread 0x1c81ea thru 0x1cdba7 (0x59be bytes) 000149[1] 1766413191.594806 mapwrite 0x929995 thru 0x932f74 (0x95e0 bytes) 000151[1] 1766413191.768682 read 0x4ca7a7 thru 0x4cbb66 (0x13c0 bytes) 000153[1] 1766413191.774704 write 0x4c5262 thru 0x4c963e (0x43dd bytes) 000157[1] 1766413191.872514 trunc from 0x932f75 to 0x787237 (0x1abd3d bytes) 000162[0] 1766413191.936314 mapwrite 0x18a213 thru 0x19009e (0x5e8c bytes) 000164[1] 1766413192.065073 write 0x577be2 thru 0x5792bc (0x16db bytes) 000166[0] 1766413192.123804 write 0x21cb97 thru 0x22ba1a (0xee84 bytes) 000167[1] 1766413192.170237 write 0x7a4a76 thru 0x7ac0e7 (0x7672 bytes) 000168[0] 1766413192.215396 read 0x13ac48 thru 0x142c04 (0x7fbd bytes) 000170[0] 1766413192.297564 mapwrite 0x4b2d11 thru 0x4be820 (0xbb10 bytes) 000171[0] 1766413192.488328 trunc from 0x7ac0e8 to 0x89d362 (0xf127b bytes) 000172[0] 1766413192.515076 trunc from 0x89d363 to 0x5520d0 (0x34b292 bytes) 000173[1] 1766413192.533168 read 0x047045 thru 0x050fbd (0x9f79 bytes) 000174[0] 1766413192.576286 mapread 0x4d9d91 thru 0x4e5862 (0xbad2 bytes) 000176[0] 1766413192.600016 mapread 0x3691e4 thru 0x372684 (0x94a1 bytes) 000177[0] 1766413192.647742 trunc from 0x5520d1 to 0x39bbd7 (0x1b64f9 bytes) 000178[0] 1766413192.679238 mapwrite 0x958ec4 thru 0x961fec (0x9129 bytes) 000179[0] 1766413192.752914 write 0x15743f thru 0x157967 (0x529 bytes) 000181[0] 1766413192.780302 mapread 0x2e1467 thru 0x2eb6cb (0xa265 bytes) 000182[1] 1766413192.822087 mapwrite 0x292dc0 thru 0x2a13ca (0xe60b bytes) 000183[0] 1766413193.003949 trunc from 0x961fed to 0x8f9b61 (0x6848b bytes) 000185[1] 1766413193.039512 read 0x5d0f40 thru 0x5de894 (0xd955 bytes) 000189[0] 1766413193.149850 trunc from 0x8f9b62 to 0x1a7f07 (0x751c5a bytes) 000190[1] 1766413193.183717 trunc from 0x1a7f08 to 0x23b630 (0x93729 bytes) 000194[0] 1766413193.239781 write 0x9c55c8 thru 0x9c9032 (0x3a6b bytes) 000195[1] 1766413193.282045 mapread 0x8764de thru 0x885eb7 (0xf9da bytes) 000196[1] 1766413193.332610 read 0x3aa60f thru 0x3b2b2c (0x851e bytes) 000198[0] 1766413193.386589 mapread 0x206a96 thru 0x207724 (0xc8f bytes) 000199[0] 1766413193.476408 mapread 0x5e0627 thru 0x5e2cc8 (0x26a2 bytes) 000201[1] 1766413193.549844 read 0x316a3a thru 0x318a39 (0x2000 bytes) 000203[1] 1766413193.556418 read 0x47beb1 thru 0x4869e3 (0xab33 bytes) 000205[1] 1766413193.575059 read 0x3caa87 thru 0x3d9145 (0xe6bf bytes) 000206[1] 1766413193.586735 mapwrite 0x9fac84 thru 0x9fc113 (0x1490 bytes) 000211[1] 1766413193.774653 trunc from 0x9fc114 to 0x57eaeb (0x47d628 bytes) 000213[0] 1766413193.822782 mapread 0x316d62 thru 0x31a91e (0x3bbd bytes) 000218[1] 1766413193.951326 write 0x0c1c1f thru 0x0c4c3d (0x301f bytes) 000219[1] 1766413193.982164 trunc from 0x57eaec to 0x7c575b (0x246c70 bytes) 000222[1] 1766413194.087325 trunc from 0x7c575c to 0x35c48e (0x4692cd bytes) 000223[0] 1766413194.140961 read 0x120b18 thru 0x1263fc (0x58e5 bytes) 000224[0] 1766413194.210715 mapwrite 0x5013a5 thru 0x50d567 (0xc1c3 bytes) 000225[0] 1766413194.468194 trunc from 0x50d568 to 0x445df7 (0xc7770 bytes) 000226[1] 1766413194.491204 trunc from 0x445df8 to 0x607570 (0x1c1779 bytes) 000232[1] 1766413194.643650 mapwrite 0x175ed7 thru 0x179f73 (0x409d bytes) 000233[1] 1766413194.699852 mapwrite 0x627fb3 thru 0x62e103 (0x6151 bytes) 000234[0] 1766413194.807402 mapread 0x4234e6 thru 0x428622 (0x513d bytes) 000235[0] 1766413194.870081 write 0x5c834c thru 0x5d7521 (0xf1d6 bytes) 000238[1] 1766413194.936068 mapread 0x2cf54a thru 0x2de1d0 (0xec87 bytes) 000239[0] 1766413195.007892 read 0x23c25a thru 0x24035e (0x4105 bytes) 000240[0] 1766413195.040660 write 0x1d1f0b thru 0x1d8e09 (0x6eff bytes) 000246[1] 1766413195.186079 mapread 0x26fde1 thru 0x27c8cf (0xcaef bytes) 000249[0] 1766413195.237466 trunc from 0x62e104 to 0x5f8303 (0x35e00 bytes) 000250[1] 1766413195.268590 read 0x100033 thru 0x102db4 (0x2d82 bytes) 000252[0] 1766413195.290839 mapread 0x16c21e thru 0x179c70 (0xda53 bytes) 000255[1] 1766413195.383769 read 0x403dbc thru 0x4095df (0x5824 bytes) 000256[1] 1766413195.448799 mapwrite 0x7b1080 thru 0x7b5c08 (0x4b89 bytes) 000257[1] 1766413195.666825 write 0x9048af thru 0x905e49 (0x159b bytes) 000259[0] 1766413195.709677 trunc from 0x905e4a to 0x83ad7f (0xcb0ca bytes) 000260[1] 1766413195.758449 write 0x345049 thru 0x3479bd (0x2975 bytes) 000261[1] 1766413195.792057 read 0x800de1 thru 0x801558 (0x778 bytes) 000262[0] 1766413195.813617 trunc from 0x83ad80 to 0x832c2a (0x8155 bytes) 000263[1] 1766413195.845094 trunc from 0x832c2b to 0x92182a (0xeec00 bytes) 000264[0] 1766413195.914937 write 0x6ec229 thru 0x6f5dce (0x9ba6 bytes) 000266[1] 1766413195.954936 read 0x46f880 thru 0x476355 (0x6ad6 bytes) 000272[1] 1766413196.007924 mapwrite 0x6a7257 thru 0x6b46d9 (0xd483 bytes) 000274[1] 1766413196.142076 mapwrite 0x897035 thru 0x8a33c6 (0xc392 bytes) 000276[0] 1766413196.250906 mapwrite 0x46c608 thru 0x473f72 (0x796b bytes) 000278[0] 1766413196.386196 trunc from 0x92182b to 0x08a094 (0x897796 bytes) 000279[0] 1766413196.435047 write 0x09716e thru 0x0a27a3 (0xb636 bytes) truncating to largest ever: 0x95cdbf 000280[1] 1766413196.436288 trunc from 0x0a27a4 to 0x95cdbe (0x8ba61b bytes) 000283[0] 1766413196.571630 read 0x272d5d thru 0x27f2ed (0xc591 bytes) 000284[1] 1766413196.599071 read 0x690859 thru 0x692dfb (0x25a3 bytes) 000287[0] 1766413196.649742 mapwrite 0x49c00d thru 0x4a4c37 (0x8c2b bytes) 000288[0] 1766413196.735231 trunc from 0x95cdbf to 0x1da33e (0x782a80 bytes) 000289[1] 1766413196.789817 trunc from 0x1da33f to 0x5236da (0x34939c bytes) 000290[1] 1766413196.839559 write 0x9c7b1d thru 0x9ce654 (0x6b38 bytes) 000292[0] 1766413196.880259 write 0x4e712f thru 0x4edc44 (0x6b16 bytes) 000297[1] 1766413196.992832 write 0x33240b thru 0x3349d2 (0x25c8 bytes) 000299[0] 1766413197.034073 trunc from 0x9ce655 to 0x0244f5 (0x9aa15f bytes) 000300[1] 1766413197.112492 mapread 0x011e37 thru 0x01fd60 (0xdf2a bytes) 000302[0] 1766413197.256193 read 0x01ef8a thru 0x020afa (0x1b71 bytes) 000303[1] 1766413197.283889 mapread 0x00aba3 thru 0x00ef9c (0x43fa bytes) 000305[1] 1766413197.310697 read 0x020de1 thru 0x0244f5 (0x3715 bytes) 000308[1] 1766413197.344429 trunc from 0x0244f6 to 0x05b0d1 (0x36bdc bytes) 000309[1] 1766413197.386749 trunc from 0x05b0d2 to 0x66d0b1 (0x611fe0 bytes) 000310[0] 1766413197.432986 mapwrite 0x24b86a thru 0x25a918 (0xf0af bytes) 000312[0] 1766413197.589855 trunc from 0x66d0b2 to 0x2edd7d (0x37f334 bytes) 000315[1] 1766413197.646265 read 0x12d082 thru 0x13c96f (0xf8ee bytes) 000316[0] 1766413197.736050 mapwrite 0x955d17 thru 0x965c32 (0xff1c bytes) 000317[1] 1766413197.963435 mapread 0x74add2 thru 0x757df6 (0xd025 bytes) 000319[1] 1766413198.059748 trunc from 0x965c33 to 0x0ab838 (0x8ba3fa bytes) 000321[1] 1766413198.172978 write 0x46df4d thru 0x47695f (0x8a13 bytes) 000322[0] 1766413198.210048 write 0x271e99 thru 0x27f8b2 (0xda1a bytes) 000325[0] 1766413198.331717 write 0x11f6f4 thru 0x127af7 (0x8404 bytes) 000329[0] 1766413198.421753 trunc from 0x476960 to 0x8322ac (0x3bb94d bytes) 000330[0] 1766413198.470086 write 0x431b51 thru 0x434705 (0x2bb5 bytes) 000333[1] 1766413198.499595 mapread 0x595b1c thru 0x596404 (0x8e9 bytes) 000334[1] 1766413198.541008 trunc from 0x8322ad to 0x46509e (0x3cd20e bytes) 000335[0] 1766413198.638032 write 0x234b02 thru 0x236771 (0x1c70 bytes) 000336[1] 1766413198.674442 trunc from 0x46509f to 0x8a5643 (0x4405a5 bytes) 000337[1] 1766413198.712492 mapread 0x0f162d thru 0x0fa0f4 (0x8ac8 bytes) 000339[1] 1766413198.803426 write 0x56cc5c thru 0x5772a1 (0xa646 bytes) 000340[0] 1766413198.871596 mapwrite 0x02a7a0 thru 0x02ed37 (0x4598 bytes) 000342[0] 1766413199.021718 mapread 0x328648 thru 0x32b698 (0x3051 bytes) 000344[1] 1766413199.105054 read 0x585df7 thru 0x58dc57 (0x7e61 bytes) 000346[0] 1766413199.144331 trunc from 0x8a5644 to 0x0fa330 (0x7ab313 bytes) 000351[1] 1766413199.239932 mapread 0x0d2cc1 thru 0x0d49ec (0x1d2c bytes) 000356[0] 1766413199.349217 trunc from 0x0fa331 to 0x584799 (0x48a469 bytes) 000358[0] 1766413199.406523 mapread 0x338eca thru 0x342203 (0x933a bytes) 000359[1] 1766413199.485019 read 0x37c6ef thru 0x38a918 (0xe22a bytes) 000360[0] 1766413199.538611 mapwrite 0x4171f4 thru 0x41de2e (0x6c3b bytes) 000362[0] 1766413199.654037 mapwrite 0x45b56f thru 0x46aeed (0xf97f bytes) 000363[1] 1766413199.782792 trunc from 0x58479a to 0x8afd97 (0x32b5fe bytes) 000364[1] 1766413199.862993 mapwrite 0x9b86e5 thru 0x9c5250 (0xcb6c bytes) 000365[1] 1766413199.951864 write 0x17eda1 thru 0x18e1a1 (0xf401 bytes) 000368[1] 1766413199.992405 mapread 0x42d651 thru 0x43b7cb (0xe17b bytes) 000369[0] 1766413200.052763 mapread 0x119932 thru 0x122b5a (0x9229 bytes) 000370[1] 1766413200.192560 mapwrite 0x4a7cf5 thru 0x4ad991 (0x5c9d bytes) 000371[1] 1766413200.272776 trunc from 0x9c5251 to 0x26733c (0x75df14 bytes) 000372[1] 1766413200.307300 trunc from 0x26733d to 0x57fcc4 (0x318988 bytes) 000376[1] 1766413200.393441 trunc from 0x57fcc5 to 0x50f78a (0x7053a bytes) 000377[0] 1766413200.426560 mapread 0x41372e thru 0x41d1d2 (0x9aa5 bytes) 000379[1] 1766413200.584837 mapwrite 0x23c73c thru 0x23ce58 (0x71d bytes) 000380[1] 1766413200.655552 write 0x2384ba thru 0x23f8ce (0x7415 bytes) 000384[0] 1766413200.693582 mapread 0x2ce55e thru 0x2d3634 (0x50d7 bytes) 000387[1] 1766413200.753879 trunc from 0x50f78b to 0x3ba162 (0x155628 bytes) 000392[1] 1766413200.873621 read 0x23a5e6 thru 0x243f91 (0x99ac bytes) 000393[1] 1766413200.978511 mapread 0x0e3424 thru 0x0ed8dc (0xa4b9 bytes) 000394[1] 1766413200.995756 mapread 0x358729 thru 0x35916b (0xa43 bytes) 000398[1] 1766413201.050827 write 0x9c04f4 thru 0x9ce94b (0xe458 bytes) 000399[1] 1766413201.053748 write 0x4e1a59 thru 0x4e6cf1 (0x5299 bytes) 000401[1] 1766413201.154966 read 0x7807be thru 0x78de84 (0xd6c7 bytes) 000404[1] 1766413201.206879 trunc from 0x9ce94c to 0x08098f (0x94dfbc bytes) 000405[1] 1766413201.319687 mapread 0x03dd43 thru 0x046dbe (0x907c bytes) 000406[0] 1766413201.354412 write 0x17cb9d thru 0x17e17c (0x15e0 bytes) 000407[0] 1766413201.395607 trunc from 0x17e17d to 0x40f8ff (0x291783 bytes) 000408[1] 1766413201.482275 mapwrite 0x4c9925 thru 0x4cdab1 (0x418d bytes) 000412[0] 1766413201.712931 mapread 0x135394 thru 0x142884 (0xd4f1 bytes) 000415[0] 1766413201.999969 mapread 0x2e705f thru 0x2f0a7a (0x9a1c bytes) 000417[1] 1766413202.161705 trunc from 0x4cdab2 to 0x34bb6c (0x181f45 bytes) 000421[1] 1766413202.283554 write 0x2d5f74 thru 0x2d9051 (0x30de bytes) 000422[1] 1766413202.293922 trunc from 0x34bb6d to 0x8d8511 (0x58c9a5 bytes) 000423[0] 1766413202.323042 read 0x2989b9 thru 0x29d695 (0x4cdd bytes) 000427[1] 1766413202.412433 read 0x8a6b5c thru 0x8ab0aa (0x454f bytes) truncating to largest ever: 0x994ce9 000430[1] 1766413202.475933 trunc from 0x8d8512 to 0x994ce8 (0xbc7d7 bytes) 000431[0] 1766413202.557962 mapwrite 0x9db01e thru 0x9de8aa (0x388d bytes) 000432[0] 1766413202.806624 read 0x4db528 thru 0x4dc543 (0x101c bytes) 000433[0] 1766413202.814213 write 0x21c1d9 thru 0x21f652 (0x347a bytes) 000436[1] 1766413202.891950 trunc from 0x9de8ab to 0x425710 (0x5b919a bytes) 000437[0] 1766413203.003602 mapread 0x2d866c thru 0x2e66bb (0xe050 bytes) 000439[1] 1766413203.085157 mapwrite 0x368162 thru 0x37365f (0xb4fe bytes) 000440[0] 1766413203.209963 mapwrite 0x2ddb46 thru 0x2ed499 (0xf954 bytes) 000442[1] 1766413203.413761 mapread 0x30e94b thru 0x3156ac (0x6d62 bytes) 000444[0] 1766413203.506166 mapwrite 0x2b26e9 thru 0x2bc96f (0xa287 bytes) 000446[1] 1766413203.722851 write 0x0fa5d3 thru 0x1064e5 (0xbf13 bytes) 000450[1] 1766413203.848779 trunc from 0x425711 to 0x130f55 (0x2f47bb bytes) 000454[0] 1766413203.916771 write 0x9e2fba thru 0x9ed7ce (0xa815 bytes) 000455[1] 1766413203.955169 read 0x1029c1 thru 0x10902e (0x666e bytes) 000462[0] 1766413204.050695 read 0x27e6c6 thru 0x28776e (0x90a9 bytes) 000469[1] 1766413204.092296 trunc from 0x9ed7cf to 0x8a58e7 (0x147ee7 bytes) 000476[1] 1766413204.262424 write 0x4eea0d thru 0x4f9916 (0xaf0a bytes) 000481[1] 1766413204.321805 mapwrite 0x0a5964 thru 0x0b08af (0xaf4c bytes) 000483[1] 1766413204.450921 mapread 0x796c37 thru 0x7a55b0 (0xe97a bytes) 000487[1] 1766413204.520769 write 0x37ba18 thru 0x37bb41 (0x12a bytes) 000488[0] 1766413204.527071 write 0x1a33cb thru 0x1a5858 (0x248e bytes) 000489[1] 1766413204.570476 mapread 0x6b776c thru 0x6bd1e9 (0x5a7e bytes) 000491[0] 1766413204.633675 trunc from 0x8a58e8 to 0x107378 (0x79e56f bytes) 000493[0] 1766413204.717463 trunc from 0x107379 to 0x0abfa2 (0x5b3d6 bytes) 000494[0] 1766413204.755573 mapread 0x01fd71 thru 0x024127 (0x43b7 bytes) 000496[1] 1766413204.798432 read 0x0229f4 thru 0x02c3d9 (0x99e6 bytes) 000497[0] 1766413204.864418 trunc from 0x0abfa3 to 0x3b4086 (0x3080e4 bytes) 000499[1] 1766413204.935924 trunc from 0x3b4087 to 0x09f4fd (0x314b89 bytes) 000500[1] 1766413204.979034 trunc from 0x09f4fe to 0x842b7c (0x7a367f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3205 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000008[1] 1766413205.784510 write 0x46e000 thru 0x47b48a (0xd48b bytes) truncating to largest ever: 0x77e213 000009[1] 1766413205.839430 trunc from 0x47b48b to 0x77e212 (0x302d88 bytes) 000012[1] 1766413205.910517 write 0x75a000 thru 0x75a107 (0x108 bytes) 000013[1] 1766413205.918352 mapread 0x53c000 thru 0x549002 (0xd003 bytes) 000014[0] 1766413206.017223 write 0x8a0000 thru 0x8ad841 (0xd842 bytes) 000015[1] 1766413206.081722 mapwrite 0x01e000 thru 0x02a5f3 (0xc5f4 bytes) 000018[1] 1766413206.258832 read 0x62a000 thru 0x638efa (0xeefb bytes) 000019[1] 1766413206.267735 mapread 0x33f000 thru 0x34078f (0x1790 bytes) 000021[0] 1766413206.295895 trunc from 0x8ad842 to 0x1cde88 (0x6df9b9 bytes) 000022[0] 1766413206.336802 read 0x17b000 thru 0x17d394 (0x2395 bytes) 000025[1] 1766413206.365350 trunc from 0x1cde89 to 0x02c61b (0x1a186d bytes) 000026[1] 1766413206.411802 write 0x18c000 thru 0x199a5e (0xda5f bytes) 000027[0] 1766413206.431715 mapread 0x127000 thru 0x135c79 (0xec7a bytes) 000028[0] 1766413206.474845 read 0x0a9000 thru 0x0b7081 (0xe082 bytes) 000029[0] 1766413206.519364 write 0x066000 thru 0x0702e2 (0xa2e3 bytes) 000039[0] 1766413206.757104 trunc from 0x199a5f to 0x6df376 (0x545918 bytes) 000040[0] 1766413206.824804 trunc from 0x6df377 to 0x480346 (0x25f030 bytes) 000041[1] 1766413206.840979 mapread 0x35f000 thru 0x3645eb (0x55ec bytes) 000043[0] 1766413206.887532 read 0x255000 thru 0x257243 (0x2244 bytes) 000046[1] 1766413206.893854 trunc from 0x480347 to 0x5a4897 (0x124551 bytes) 000048[1] 1766413206.923526 write 0x50e000 thru 0x51b8be (0xd8bf bytes) 000049[0] 1766413206.932731 read 0x146000 thru 0x14cd1b (0x6d1c bytes) 000050[0] 1766413206.978473 mapread 0x020000 thru 0x02c166 (0xc167 bytes) 000051[1] 1766413207.006712 mapread 0x445000 thru 0x4484e2 (0x34e3 bytes) 000055[1] 1766413207.043187 trunc from 0x5a4898 to 0x4c5d26 (0xdeb71 bytes) 000056[1] 1766413207.103752 trunc from 0x4c5d27 to 0x0a6e2c (0x41eefa bytes) 000058[0] 1766413207.187890 read 0x0a3000 thru 0x0a67f1 (0x37f2 bytes) 000063[1] 1766413207.245253 write 0x2fc000 thru 0x304139 (0x813a bytes) 000066[1] 1766413207.292789 write 0x488000 thru 0x497428 (0xf429 bytes) 000067[0] 1766413207.298904 trunc from 0x497429 to 0x5d339c (0x13bf74 bytes) 000068[0] 1766413207.364850 mapread 0x07c000 thru 0x0823d1 (0x63d2 bytes) 000069[1] 1766413207.479359 trunc from 0x5d339d to 0x71f2af (0x14bf13 bytes) 000070[1] 1766413207.523521 mapwrite 0x047000 thru 0x04ec74 (0x7c75 bytes) 000073[0] 1766413207.711344 trunc from 0x71f2b0 to 0x345e5c (0x3d9453 bytes) 000075[1] 1766413207.792680 mapwrite 0x2f9000 thru 0x2feb0d (0x5b0e bytes) 000078[0] 1766413207.945909 write 0x353000 thru 0x35e360 (0xb361 bytes) 000079[0] 1766413207.967232 mapwrite 0x519000 thru 0x524e40 (0xbe41 bytes) 000081[1] 1766413208.124934 write 0x702000 thru 0x707124 (0x5125 bytes) 000086[0] 1766413208.170776 mapread 0x349000 thru 0x349395 (0x396 bytes) 000087[1] 1766413208.205967 mapread 0x6db000 thru 0x6e0cab (0x5cac bytes) 000088[0] 1766413208.242957 mapwrite 0x106000 thru 0x10662f (0x630 bytes) 000089[1] 1766413208.360835 write 0x183000 thru 0x189c4a (0x6c4b bytes) 000091[1] 1766413208.424201 mapwrite 0x588000 thru 0x58ee10 (0x6e11 bytes) 000092[1] 1766413208.511976 read 0x271000 thru 0x275862 (0x4863 bytes) 000093[0] 1766413208.543548 read 0x0c7000 thru 0x0d59aa (0xe9ab bytes) 000094[1] 1766413208.561091 write 0x5e7000 thru 0x5f361f (0xc620 bytes) 000096[1] 1766413208.625057 write 0x83e000 thru 0x84d939 (0xf93a bytes) 000097[1] 1766413208.644820 write 0x0e1000 thru 0x0e645a (0x545b bytes) 000100[0] 1766413208.679020 write 0x912000 thru 0x91e7ba (0xc7bb bytes) 000101[0] 1766413208.713833 mapread 0x1a4000 thru 0x1a66f7 (0x26f8 bytes) 000105[0] 1766413208.767681 write 0x6a7000 thru 0x6b5b2b (0xeb2c bytes) truncating to largest ever: 0x7c9064 000106[0] 1766413208.799104 trunc from 0x91e7bb to 0x7c9063 (0x155757 bytes) 000108[1] 1766413208.838763 mapread 0x646000 thru 0x64b6c2 (0x56c3 bytes) 000109[1] 1766413208.906079 trunc from 0x7c9064 to 0x70e6b4 (0xba9af bytes) 000110[0] 1766413208.971317 mapwrite 0x194000 thru 0x1a0859 (0xc85a bytes) 000112[0] 1766413209.077791 mapread 0x677000 thru 0x67cd48 (0x5d49 bytes) 000116[0] 1766413209.134708 mapwrite 0x464000 thru 0x466835 (0x2836 bytes) 000121[1] 1766413209.236415 write 0x1ad000 thru 0x1b1ec5 (0x4ec6 bytes) 000123[1] 1766413209.254888 mapwrite 0x9bf000 thru 0x9c4ac8 (0x5ac9 bytes) 000125[1] 1766413209.419995 write 0x2b9000 thru 0x2c21fb (0x91fc bytes) 000129[0] 1766413209.459580 trunc from 0x9c4ac9 to 0x4051d1 (0x5bf8f7 bytes) truncating to largest ever: 0x953005 000131[1] 1766413209.509893 trunc from 0x4051d2 to 0x953004 (0x54de33 bytes) 000137[1] 1766413209.693851 read 0x63a000 thru 0x644e9f (0xaea0 bytes) 000138[0] 1766413209.723164 trunc from 0x953005 to 0x49bf93 (0x4b7071 bytes) 000140[0] 1766413209.750607 trunc from 0x49bf94 to 0x6a1276 (0x2052e3 bytes) 000142[0] 1766413209.789014 write 0x1f1000 thru 0x1f6e0a (0x5e0b bytes) 000144[1] 1766413209.805236 read 0x670000 thru 0x67d2f4 (0xd2f5 bytes) 000147[0] 1766413209.859889 write 0x905000 thru 0x914b57 (0xfb58 bytes) 000148[0] 1766413209.909941 read 0x8e3000 thru 0x8eadf7 (0x7df8 bytes) 000150[0] 1766413209.979579 trunc from 0x914b58 to 0x311759 (0x6033fe bytes) 000151[0] 1766413210.042527 write 0x987000 thru 0x98f31b (0x831c bytes) 000152[1] 1766413210.061647 trunc from 0x98f31c to 0x64917f (0x34619c bytes) 000154[0] 1766413210.143818 read 0x1b8000 thru 0x1bac26 (0x2c27 bytes) 000155[0] 1766413210.169322 write 0x17b000 thru 0x180611 (0x5612 bytes) 000157[1] 1766413210.187055 trunc from 0x649180 to 0x015516 (0x633c69 bytes) 000158[1] 1766413210.276525 mapread 0x00d000 thru 0x00d521 (0x522 bytes) 000162[1] 1766413210.325029 write 0x670000 thru 0x676e45 (0x6e46 bytes) 000163[0] 1766413210.341870 write 0x55e000 thru 0x560093 (0x2094 bytes) 000164[0] 1766413210.382337 read 0x4d2000 thru 0x4d8868 (0x6869 bytes) 000165[0] 1766413210.393666 read 0x4b9000 thru 0x4be9d0 (0x59d1 bytes) 000166[1] 1766413210.403730 trunc from 0x676e46 to 0x2ce9a4 (0x3a84a1 bytes) 000168[1] 1766413210.446625 read 0x16b000 thru 0x16fe85 (0x4e86 bytes) 000172[1] 1766413210.553821 read 0x0aa000 thru 0x0ac700 (0x2701 bytes) 000174[0] 1766413210.598287 trunc from 0x2ce9a5 to 0x88dbe3 (0x5bf23f bytes) 000178[1] 1766413210.692457 mapwrite 0x1e9000 thru 0x1ea825 (0x1826 bytes) 000181[0] 1766413210.827933 read 0x42c000 thru 0x43499a (0x899b bytes) 000182[0] 1766413210.849085 mapread 0x0ad000 thru 0x0bcdf0 (0xfdf1 bytes) 000184[0] 1766413210.879832 mapwrite 0x78c000 thru 0x78f81e (0x381f bytes) 000185[0] 1766413210.955147 mapwrite 0x5b6000 thru 0x5bf81f (0x9820 bytes) 000186[0] 1766413211.036906 write 0x203000 thru 0x206fa6 (0x3fa7 bytes) 000187[1] 1766413211.056291 read 0x2da000 thru 0x2dece2 (0x4ce3 bytes) 000188[0] 1766413211.091679 write 0x2ea000 thru 0x2eefc6 (0x4fc7 bytes) 000189[0] 1766413211.119209 mapwrite 0x623000 thru 0x62f882 (0xc883 bytes) 000190[1] 1766413211.225523 write 0x9ca000 thru 0x9cb2e7 (0x12e8 bytes) 000197[0] 1766413211.281400 trunc from 0x9cb2e8 to 0x16a125 (0x8611c2 bytes) 000199[0] 1766413211.337567 read 0x0e4000 thru 0x0ec595 (0x8596 bytes) 000201[1] 1766413211.407958 write 0x4d4000 thru 0x4d5c79 (0x1c7a bytes) 000204[0] 1766413211.445032 trunc from 0x4d5c7a to 0x370f4d (0x164d2c bytes) 000205[1] 1766413211.516200 trunc from 0x370f4e to 0x3fffa4 (0x8f057 bytes) 000206[0] 1766413211.551674 write 0x90f000 thru 0x914985 (0x5986 bytes) 000208[0] 1766413211.615793 read 0x4d9000 thru 0x4e51d3 (0xc1d4 bytes) 000211[1] 1766413211.666672 mapwrite 0x9ea000 thru 0x9ea788 (0x789 bytes) 000212[1] 1766413211.847234 mapwrite 0x021000 thru 0x028085 (0x7086 bytes) 000214[1] 1766413212.011149 mapread 0x1b1000 thru 0x1bd544 (0xc545 bytes) 000215[1] 1766413212.062271 write 0x556000 thru 0x565879 (0xf87a bytes) 000216[1] 1766413212.083958 trunc from 0x9ea789 to 0x6c99a0 (0x320de8 bytes) 000218[1] 1766413212.123538 read 0x162000 thru 0x164616 (0x2617 bytes) 000222[0] 1766413212.159005 mapread 0x1bf000 thru 0x1ce16c (0xf16d bytes) 000223[0] 1766413212.215372 mapwrite 0x41c000 thru 0x42779a (0xb79b bytes) 000224[1] 1766413212.297909 trunc from 0x6c99a1 to 0x71a5d9 (0x50c39 bytes) 000225[1] 1766413212.330235 write 0x31a000 thru 0x3204e7 (0x64e8 bytes) 000226[1] 1766413212.335191 mapread 0x29d000 thru 0x2ab4ca (0xe4cb bytes) 000232[1] 1766413212.503326 mapwrite 0x143000 thru 0x148817 (0x5818 bytes) 000235[1] 1766413212.626867 mapwrite 0x996000 thru 0x9a3038 (0xd039 bytes) 000238[1] 1766413212.777373 mapwrite 0x4f8000 thru 0x4fc2c9 (0x42ca bytes) 000245[1] 1766413213.028567 mapwrite 0x84d000 thru 0x84d21a (0x21b bytes) 000246[1] 1766413213.139554 read 0x82e000 thru 0x831a70 (0x3a71 bytes) 000249[0] 1766413213.179305 mapwrite 0x90f000 thru 0x912780 (0x3781 bytes) 000252[1] 1766413213.347660 mapread 0x5f5000 thru 0x5fd4b6 (0x84b7 bytes) 000254[1] 1766413213.380740 trunc from 0x9a3039 to 0x76ba9d (0x23759b bytes) 000256[0] 1766413213.438616 read 0x044000 thru 0x048afd (0x4afe bytes) 000257[0] 1766413213.461885 trunc from 0x76ba9e to 0x25f03d (0x50ca60 bytes) 000258[0] 1766413213.491041 read 0x1ba000 thru 0x1c2acb (0x8acc bytes) 000259[1] 1766413213.567160 mapwrite 0x32d000 thru 0x33b436 (0xe437 bytes) 000260[1] 1766413213.671273 trunc from 0x33b437 to 0x867fa3 (0x52cb6d bytes) 000262[1] 1766413213.706996 mapread 0x6e2000 thru 0x6e4b2f (0x2b30 bytes) 000266[0] 1766413213.766508 mapread 0x3fa000 thru 0x40779b (0xd79c bytes) 000267[1] 1766413213.841279 mapwrite 0x208000 thru 0x20aef1 (0x2ef2 bytes) 000268[1] 1766413213.941859 mapread 0x6a0000 thru 0x6aea26 (0xea27 bytes) 000269[0] 1766413213.994046 trunc from 0x867fa4 to 0x49bc6e (0x3cc335 bytes) 000270[0] 1766413214.072226 read 0x292000 thru 0x29d69a (0xb69b bytes) 000271[0] 1766413214.085422 trunc from 0x49bc6f to 0x66da7d (0x1d1e0f bytes) 000276[0] 1766413214.280624 read 0x43a000 thru 0x43b471 (0x1472 bytes) 000277[0] 1766413214.310177 trunc from 0x66da7e to 0x2a9767 (0x3c4316 bytes) 000279[1] 1766413214.387022 write 0x223000 thru 0x22fd84 (0xcd85 bytes) 000280[0] 1766413214.427975 mapwrite 0x5a3000 thru 0x5a99b2 (0x69b3 bytes) 000282[1] 1766413214.533289 read 0x4f6000 thru 0x4f62f5 (0x2f6 bytes) 000283[0] 1766413214.624156 read 0x3c4000 thru 0x3cd2b1 (0x92b2 bytes) 000284[1] 1766413214.657877 read 0x4bc000 thru 0x4bfe82 (0x3e83 bytes) 000286[1] 1766413214.663269 mapwrite 0x819000 thru 0x819cf4 (0xcf5 bytes) 000288[1] 1766413214.789893 trunc from 0x819cf5 to 0x77a758 (0x9f59c bytes) 000290[0] 1766413214.822945 mapwrite 0x748000 thru 0x74d3d2 (0x53d3 bytes) 000291[0] 1766413214.961814 read 0x462000 thru 0x46783f (0x5840 bytes) 000292[0] 1766413215.017794 read 0x71f000 thru 0x72573b (0x673c bytes) 000293[0] 1766413215.071438 write 0x8a6000 thru 0x8a8618 (0x2619 bytes) 000298[0] 1766413215.138212 mapread 0x0b6000 thru 0x0b65ce (0x5cf bytes) 000304[0] 1766413215.172344 mapwrite 0x882000 thru 0x88a3b1 (0x83b2 bytes) 000305[1] 1766413215.248918 write 0x994000 thru 0x9a0441 (0xc442 bytes) 000306[1] 1766413215.261099 mapwrite 0x974000 thru 0x976ea0 (0x2ea1 bytes) 000307[1] 1766413215.388454 trunc from 0x9a0442 to 0x5a6cbd (0x3f9784 bytes) 000309[1] 1766413215.407412 write 0x052000 thru 0x05eb15 (0xcb16 bytes) 000314[0] 1766413215.483286 mapwrite 0x6ab000 thru 0x6affba (0x4fbb bytes) 000315[1] 1766413215.605888 write 0x111000 thru 0x1114d2 (0x4d3 bytes) 000316[0] 1766413215.638031 write 0x364000 thru 0x367e1c (0x3e1d bytes) 000318[1] 1766413215.672790 mapread 0x34a000 thru 0x3536b8 (0x96b9 bytes) 000323[1] 1766413215.751524 read 0x167000 thru 0x16c343 (0x5344 bytes) 000326[0] 1766413215.759166 write 0x94a000 thru 0x952a31 (0x8a32 bytes) 000327[1] 1766413215.777887 mapread 0x414000 thru 0x420b41 (0xcb42 bytes) 000328[1] 1766413215.827708 mapread 0x6a9000 thru 0x6b20b6 (0x90b7 bytes) 000329[0] 1766413215.850855 write 0x91f000 thru 0x92d8fd (0xe8fe bytes) 000330[0] 1766413215.916578 mapwrite 0x25f000 thru 0x2671ca (0x81cb bytes) 000331[0] 1766413216.014994 mapwrite 0x9cf000 thru 0x9decce (0xfccf bytes) 000333[0] 1766413216.196737 read 0x6d5000 thru 0x6dbe95 (0x6e96 bytes) 000335[0] 1766413216.210203 read 0x538000 thru 0x5460ca (0xe0cb bytes) 000337[1] 1766413216.248656 write 0x1a7000 thru 0x1b1c6e (0xac6f bytes) 000341[1] 1766413216.305037 mapwrite 0x8d8000 thru 0x8e0304 (0x8305 bytes) 000342[1] 1766413216.436026 write 0x81b000 thru 0x820d95 (0x5d96 bytes) truncating to largest ever: 0x9d2abc 000344[0] 1766413216.458088 trunc from 0x9deccf to 0x9d2abb (0xc213 bytes) 000345[1] 1766413216.537995 write 0x06c000 thru 0x07aace (0xeacf bytes) 000347[1] 1766413216.570372 write 0x3c9000 thru 0x3d72b5 (0xe2b6 bytes) 000350[0] 1766413216.595368 write 0x4c4000 thru 0x4ca1dd (0x61de bytes) 000356[0] 1766413216.747677 mapread 0x4c8000 thru 0x4ce83e (0x683f bytes) 000357[0] 1766413216.781904 trunc from 0x9d2abc to 0x96d0bd (0x659fe bytes) 000358[1] 1766413216.821200 trunc from 0x96d0be to 0x2f50bf (0x677ffe bytes) 000361[0] 1766413216.987227 read 0x2f2000 thru 0x2f4c97 (0x2c98 bytes) 000362[0] 1766413217.110982 read 0x280000 thru 0x287081 (0x7082 bytes) 000364[1] 1766413217.111664 mapread 0x096000 thru 0x0a1226 (0xb227 bytes) 000365[1] 1766413217.167212 write 0x2fa000 thru 0x30938b (0xf38c bytes) 000366[0] 1766413217.225585 write 0x9bd000 thru 0x9c1293 (0x4294 bytes) 000369[0] 1766413217.346428 mapread 0x6cf000 thru 0x6d709e (0x809f bytes) 000371[0] 1766413217.531987 trunc from 0x9c1294 to 0x12affb (0x896298 bytes) 000373[1] 1766413217.661360 mapread 0x02d000 thru 0x030204 (0x3205 bytes) 000376[0] 1766413217.709016 read 0x064000 thru 0x06aeb9 (0x6eba bytes) 000377[0] 1766413217.771652 read 0x0f0000 thru 0x0f821f (0x8220 bytes) 000380[0] 1766413217.779510 mapread 0x0ad000 thru 0x0b2278 (0x5279 bytes) 000381[1] 1766413217.780006 read 0x10c000 thru 0x10c147 (0x148 bytes) 000382[0] 1766413217.798276 mapwrite 0x311000 thru 0x31a65f (0x9660 bytes) 000389[1] 1766413218.115432 trunc from 0x31a660 to 0x957b2d (0x63d4ce bytes) 000391[1] 1766413218.254693 mapread 0x8c1000 thru 0x8cc118 (0xb119 bytes) 000392[1] 1766413218.390181 mapread 0x1c1000 thru 0x1cba5b (0xaa5c bytes) 000393[0] 1766413218.445886 read 0x3a6000 thru 0x3b5851 (0xf852 bytes) 000395[0] 1766413218.493666 mapread 0x339000 thru 0x341b88 (0x8b89 bytes) 000396[0] 1766413218.529513 trunc from 0x957b2e to 0x5ca50a (0x38d623 bytes) 000398[1] 1766413218.657575 trunc from 0x5ca50b to 0x80e3b3 (0x243ea9 bytes) 000399[1] 1766413218.746753 trunc from 0x80e3b4 to 0x4ffe80 (0x30e533 bytes) 000400[1] 1766413218.786477 read 0x365000 thru 0x369826 (0x4827 bytes) 000401[0] 1766413218.845092 read 0x3c0000 thru 0x3c38e7 (0x38e8 bytes) 000405[0] 1766413218.900242 write 0x093000 thru 0x09c093 (0x9094 bytes) 000407[0] 1766413218.963959 trunc from 0x4ffe81 to 0x810eda (0x31105a bytes) 000409[0] 1766413219.055722 read 0x0b4000 thru 0x0b6064 (0x2065 bytes) 000410[0] 1766413219.071884 write 0x96a000 thru 0x96e4cd (0x44ce bytes) 000415[0] 1766413219.167888 write 0x437000 thru 0x444d4d (0xdd4e bytes) 000416[0] 1766413219.185430 mapwrite 0x368000 thru 0x36e6e0 (0x66e1 bytes) 000417[1] 1766413219.330087 mapread 0x2f0000 thru 0x2f4b88 (0x4b89 bytes) 000419[1] 1766413219.401052 read 0x89d000 thru 0x8a22f3 (0x52f4 bytes) 000421[1] 1766413219.422839 trunc from 0x96e4ce to 0x2343c1 (0x73a10c bytes) 000423[0] 1766413219.504398 mapread 0x189000 thru 0x18ec33 (0x5c34 bytes) 000424[0] 1766413219.591019 mapwrite 0x1fe000 thru 0x1ff8e3 (0x18e4 bytes) 000425[1] 1766413219.714515 write 0x114000 thru 0x11ef93 (0xaf94 bytes) 000426[0] 1766413219.750718 read 0x1d0000 thru 0x1d78c7 (0x78c8 bytes) 000427[1] 1766413219.882517 mapwrite 0x456000 thru 0x46040d (0xa40e bytes) 000430[0] 1766413220.050490 write 0x50b000 thru 0x5186c1 (0xd6c2 bytes) 000433[0] 1766413220.104038 mapread 0x2c8000 thru 0x2cbad9 (0x3ada bytes) 000434[0] 1766413220.167496 trunc from 0x5186c2 to 0x7a5fa7 (0x28d8e6 bytes) 000436[0] 1766413220.198462 mapread 0x267000 thru 0x268191 (0x1192 bytes) 000442[1] 1766413220.248863 write 0x37d000 thru 0x388028 (0xb029 bytes) 000444[1] 1766413220.287682 mapread 0x62f000 thru 0x6341b0 (0x51b1 bytes) 000445[0] 1766413220.348012 read 0x29b000 thru 0x2aac70 (0xfc71 bytes) 000447[0] 1766413220.390444 mapread 0x0a8000 thru 0x0a986f (0x1870 bytes) 000449[0] 1766413220.420502 mapwrite 0x552000 thru 0x5539ca (0x19cb bytes) 000450[0] 1766413220.497674 read 0x475000 thru 0x4780db (0x30dc bytes) 000451[0] 1766413220.508621 read 0x604000 thru 0x613349 (0xf34a bytes) 000453[1] 1766413220.566411 trunc from 0x7a5fa8 to 0x3da5a9 (0x3cb9fe bytes) 000456[0] 1766413220.618205 mapwrite 0x064000 thru 0x0737b0 (0xf7b1 bytes) 000458[1] 1766413220.732490 mapread 0x15a000 thru 0x15e852 (0x4853 bytes) 000459[1] 1766413220.767805 read 0x17d000 thru 0x182e87 (0x5e88 bytes) 000464[0] 1766413220.849722 mapwrite 0x69c000 thru 0x69dba6 (0x1ba7 bytes) 000465[1] 1766413220.952915 read 0x51f000 thru 0x52d213 (0xe214 bytes) 000466[1] 1766413221.105336 mapread 0x54f000 thru 0x550de7 (0x1de8 bytes) 000467[1] 1766413221.145321 mapwrite 0x355000 thru 0x364a97 (0xfa98 bytes) 000469[0] 1766413221.409073 write 0x01b000 thru 0x023491 (0x8492 bytes) 000470[0] 1766413221.441310 read 0x60d000 thru 0x614b4e (0x7b4f bytes) 000472[1] 1766413221.510659 mapread 0x09e000 thru 0x0a8571 (0xa572 bytes) 000476[1] 1766413221.654839 mapread 0x314000 thru 0x31ff47 (0xbf48 bytes) 000478[0] 1766413221.743253 mapread 0x072000 thru 0x07668a (0x468b bytes) 000479[0] 1766413221.759051 mapread 0x4e8000 thru 0x4eb383 (0x3384 bytes) 000480[1] 1766413221.776618 mapwrite 0x9c8000 thru 0x9ceabe (0x6abf bytes) 000482[0] 1766413221.952489 write 0x366000 thru 0x371c6c (0xbc6d bytes) 000483[0] 1766413222.002725 mapread 0x84e000 thru 0x8577cf (0x97d0 bytes) 000485[1] 1766413222.028716 write 0x9dd000 thru 0x9e63eb (0x93ec bytes) 000486[1] 1766413222.053166 mapread 0x4c0000 thru 0x4c991e (0x991f bytes) 000487[0] 1766413222.113336 mapwrite 0x2a8000 thru 0x2a9e3a (0x1e3b bytes) 000488[1] 1766413222.294539 read 0x7f3000 thru 0x7fc9a3 (0x99a4 bytes) 000490[1] 1766413222.349774 read 0x86f000 thru 0x87adc3 (0xbdc4 bytes) 000491[1] 1766413222.378498 read 0x1ea000 thru 0x1ee038 (0x4039 bytes) 000492[1] 1766413222.396474 trunc from 0x9e63ec to 0x8fd5f2 (0xe8df9 bytes) 000497[1] 1766413222.523902 read 0x664000 thru 0x672345 (0xe346 bytes) All operations completed A-OK! PASS 16a (47s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:20:28 (1766413228) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.316658 s, 26.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.281765 s, 29.8 MB/s Chance of close/open is 1 in 50 Seed set to 3230 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1766413231.585062 mapwrite 0x54cf47 thru 0x54df74 (0x102e bytes) 000005[0] 1766413231.737398 mapwrite 0x2b50ab thru 0x2b66a4 (0x15fa bytes) 000006[0] 1766413231.842071 mapwrite 0x2a7b1d thru 0x2a80d8 (0x5bc bytes) truncating to largest ever: 0x8d7403 000007[1] 1766413231.896094 trunc from 0x54df75 to 0x8d7402 (0x38948e bytes) 000008[0] 1766413231.966389 write 0x318c8f thru 0x319a5b (0xdcd bytes) 000011[0] 1766413232.050687 mapwrite 0x55a601 thru 0x55b6d5 (0x10d5 bytes) 000013[1] 1766413232.149633 mapwrite 0x0ec4ca thru 0x0eddd7 (0x190e bytes) 000014[0] 1766413232.248063 mapwrite 0x59a4ae thru 0x59b884 (0x13d7 bytes) 000015[0] 1766413232.350843 write 0x4137b7 thru 0x4138a5 (0xef bytes) 000018[1] 1766413232.401036 trunc from 0x8d7403 to 0x350c6c (0x586796 bytes) 000019[0] 1766413232.440874 mapwrite 0x57133f thru 0x5716a6 (0x368 bytes) 000020[0] 1766413232.538282 write 0x9ffaa1 thru 0x9fffff (0x55f bytes) 000021[1] 1766413232.546193 write 0x2261b5 thru 0x2269cf (0x81b bytes) 000023[0] 1766413232.633186 mapread 0x14df0e thru 0x14fd55 (0x1e48 bytes) 000024[1] 1766413232.732316 mapwrite 0x155370 thru 0x15548f (0x120 bytes) 000025[0] 1766413232.845547 write 0x4f99ed thru 0x4faac8 (0x10dc bytes) 000028[0] 1766413232.939778 mapwrite 0x0cbc61 thru 0x0cce73 (0x1213 bytes) 000029[1] 1766413233.073341 write 0x820e87 thru 0x82289c (0x1a16 bytes) 000030[1] 1766413233.118840 mapread 0x177c4f thru 0x178254 (0x606 bytes) 000031[0] 1766413233.144392 read 0x75b544 thru 0x75c14e (0xc0b bytes) 000033[1] 1766413233.222792 write 0x751722 thru 0x752133 (0xa12 bytes) 000034[0] 1766413233.256194 trunc from 0xa00000 to 0x5a41b9 (0x45be46 bytes) 000035[1] 1766413233.332811 mapwrite 0x3b98c6 thru 0x3bb6d2 (0x1e0d bytes) 000037[1] 1766413233.437957 mapwrite 0x9feddf thru 0x9ff685 (0x8a7 bytes) 000042[0] 1766413233.646356 mapwrite 0x1e605f thru 0x1e6f54 (0xef6 bytes) 000043[1] 1766413233.718795 mapread 0x188ee7 thru 0x1898da (0x9f4 bytes) 000045[0] 1766413233.786929 mapwrite 0x4914fd thru 0x491d9c (0x8a0 bytes) 000051[1] 1766413234.000089 read 0x74780a thru 0x748630 (0xe27 bytes) 000052[1] 1766413234.036314 read 0x50db07 thru 0x50f84a (0x1d44 bytes) 000053[0] 1766413234.044554 read 0x7f5aa1 thru 0x7f7114 (0x1674 bytes) 000054[0] 1766413234.057332 mapread 0x3e1b1b thru 0x3e32b9 (0x179f bytes) 000057[0] 1766413234.089189 mapwrite 0x8378a1 thru 0x837ac1 (0x221 bytes) 000061[1] 1766413234.167352 write 0x8c397a thru 0x8c51af (0x1836 bytes) 000066[0] 1766413234.251436 trunc from 0x9ff686 to 0x3f53f9 (0x60a28c bytes) 000069[1] 1766413234.316459 mapread 0x0feda4 thru 0x100574 (0x17d1 bytes) 000071[0] 1766413234.393291 trunc from 0x3f53fa to 0x222526 (0x1d2ed3 bytes) 000073[0] 1766413234.470549 trunc from 0x222527 to 0x52485f (0x302339 bytes) 000074[0] 1766413234.492614 read 0x00d5cc thru 0x00f16a (0x1b9f bytes) 000075[0] 1766413234.498351 write 0x420397 thru 0x421a13 (0x167d bytes) 000077[1] 1766413234.550510 read 0x1e2d06 thru 0x1e4320 (0x161b bytes) 000078[0] 1766413234.576473 mapread 0x09225c thru 0x092b10 (0x8b5 bytes) 000079[0] 1766413234.616620 mapread 0x1a543d thru 0x1a567e (0x242 bytes) 000081[0] 1766413234.644016 trunc from 0x524860 to 0x4c66a9 (0x5e1b6 bytes) 000083[0] 1766413234.739267 read 0x206885 thru 0x207c1e (0x139a bytes) 000085[0] 1766413234.747685 mapread 0x1d3e66 thru 0x1d4d81 (0xf1c bytes) 000086[1] 1766413234.785092 write 0x0dee0a thru 0x0e037a (0x1571 bytes) 000087[1] 1766413234.821569 read 0x1cad48 thru 0x1cca1f (0x1cd8 bytes) 000090[1] 1766413234.895937 write 0x18bac4 thru 0x18cc52 (0x118f bytes) 000093[0] 1766413234.971465 mapread 0x1d29fa thru 0x1d332a (0x931 bytes) 000097[0] 1766413235.067235 write 0x98f6e1 thru 0x99116d (0x1a8d bytes) 000098[0] 1766413235.090487 mapread 0x837938 thru 0x838e99 (0x1562 bytes) 000099[1] 1766413235.167334 mapwrite 0x3cd75f thru 0x3ce7f3 (0x1095 bytes) 000101[1] 1766413235.315086 mapread 0x52eaba thru 0x53066d (0x1bb4 bytes) 000107[0] 1766413235.459859 mapwrite 0x668edd thru 0x669a30 (0xb54 bytes) 000108[1] 1766413235.687443 mapwrite 0x3339da thru 0x333f86 (0x5ad bytes) 000109[1] 1766413235.798099 mapwrite 0x4e8dd0 thru 0x4ead2f (0x1f60 bytes) 000110[0] 1766413235.948581 mapwrite 0x41d169 thru 0x41e0ca (0xf62 bytes) 000112[0] 1766413236.121858 write 0x3f49e1 thru 0x3f4f11 (0x531 bytes) 000113[1] 1766413236.186746 trunc from 0x99116e to 0x5f1a02 (0x39f76b bytes) 000115[1] 1766413236.296280 mapwrite 0x32a3a1 thru 0x32ac39 (0x899 bytes) 000116[0] 1766413236.423983 read 0x3ba53c thru 0x3bc3be (0x1e83 bytes) 000117[0] 1766413236.504172 read 0x19bb60 thru 0x19c252 (0x6f3 bytes) 000119[0] 1766413236.552654 mapwrite 0x8dee9d thru 0x8e00a1 (0x1205 bytes) 000121[0] 1766413236.697839 mapwrite 0x268d3e thru 0x269c94 (0xf57 bytes) 000122[0] 1766413236.753266 mapwrite 0x53882c thru 0x53a418 (0x1bed bytes) 000123[0] 1766413236.822481 write 0x8c7d54 thru 0x8c94a2 (0x174f bytes) 000124[1] 1766413236.832439 mapwrite 0x21449a thru 0x214bc5 (0x72c bytes) 000127[1] 1766413237.029319 trunc from 0x8e00a2 to 0x1a46f9 (0x73b9a8 bytes) 000128[1] 1766413237.086768 read 0x07cbd1 thru 0x07e991 (0x1dc1 bytes) 000129[1] 1766413237.152692 write 0x312e7c thru 0x313086 (0x20b bytes) 000130[1] 1766413237.153912 trunc from 0x313087 to 0x63ead1 (0x32ba4b bytes) 000131[0] 1766413237.186986 trunc from 0x63ead2 to 0x64c384 (0xd8b3 bytes) 000133[1] 1766413237.281643 read 0x354998 thru 0x355ecc (0x1535 bytes) 000134[0] 1766413237.313756 mapwrite 0x34bc31 thru 0x34bdf3 (0x1c3 bytes) 000135[0] 1766413237.411977 trunc from 0x64c385 to 0x82eb5f (0x1e27db bytes) 000137[1] 1766413237.474621 read 0x385f72 thru 0x386a99 (0xb28 bytes) 000139[0] 1766413237.538555 write 0x8ef34c thru 0x8f08f8 (0x15ad bytes) 000140[0] 1766413237.572219 write 0x7cdf71 thru 0x7ce8ea (0x97a bytes) 000142[0] 1766413237.603897 trunc from 0x8f08f9 to 0x3e3547 (0x50d3b1 bytes) 000143[0] 1766413237.701828 mapwrite 0x00eea5 thru 0x010ae3 (0x1c3f bytes) 000144[1] 1766413237.800084 write 0x205ea9 thru 0x2078b4 (0x1a0c bytes) 000146[0] 1766413237.855199 trunc from 0x3e3548 to 0x7b9fea (0x3d6aa3 bytes) 000148[0] 1766413237.922484 mapread 0x29a072 thru 0x29a1d3 (0x162 bytes) 000149[0] 1766413237.934550 mapwrite 0x1130fc thru 0x113916 (0x81b bytes) 000150[1] 1766413238.055422 mapread 0x1f1c9d thru 0x1f2f9d (0x1301 bytes) 000151[0] 1766413238.098573 mapread 0x557db9 thru 0x5587c5 (0xa0d bytes) 000153[1] 1766413238.172072 write 0x1c42fc thru 0x1c528a (0xf8f bytes) 000154[1] 1766413238.224524 mapread 0x16d509 thru 0x16ed87 (0x187f bytes) 000157[0] 1766413238.284474 write 0x42a96c thru 0x42b268 (0x8fd bytes) 000160[1] 1766413238.382359 mapread 0x4ad40a thru 0x4ad562 (0x159 bytes) 000161[1] 1766413238.459865 mapwrite 0x0aae40 thru 0x0ac1d8 (0x1399 bytes) 000163[1] 1766413238.701153 read 0x538580 thru 0x539996 (0x1417 bytes) 000164[1] 1766413238.708850 mapwrite 0x99c521 thru 0x99d257 (0xd37 bytes) 000165[0] 1766413238.943065 mapread 0x80a892 thru 0x80c003 (0x1772 bytes) 000166[1] 1766413239.086642 trunc from 0x99d258 to 0x33551a (0x667d3d bytes) 000172[0] 1766413239.184884 mapread 0x0f7ef6 thru 0x0f9d9b (0x1ea6 bytes) 000174[0] 1766413239.242526 mapread 0x1236ef thru 0x124dbf (0x16d1 bytes) 000176[1] 1766413239.268949 write 0x546b6a thru 0x548456 (0x18ed bytes) 000177[1] 1766413239.275667 read 0x491d5a thru 0x493a67 (0x1d0e bytes) 000181[0] 1766413239.344029 mapread 0x1b6ddb thru 0x1b89f1 (0x1c17 bytes) 000182[0] 1766413239.390873 mapread 0x3e4d51 thru 0x3e5e97 (0x1147 bytes) 000184[0] 1766413239.463940 mapwrite 0x5aa860 thru 0x5aaf79 (0x71a bytes) 000185[1] 1766413239.724107 trunc from 0x5aaf7a to 0x15fa46 (0x44b533 bytes) 000186[0] 1766413239.791593 write 0x0f2955 thru 0x0f2c87 (0x333 bytes) 000187[0] 1766413239.821772 trunc from 0x15fa47 to 0x4befe9 (0x35f5a3 bytes) 000188[0] 1766413239.860659 trunc from 0x4befea to 0x46ce80 (0x52169 bytes) 000191[0] 1766413239.939473 read 0x238056 thru 0x239874 (0x181f bytes) 000192[0] 1766413239.967646 trunc from 0x46ce81 to 0x137566 (0x33591a bytes) 000193[1] 1766413240.007484 mapwrite 0x661d87 thru 0x662a47 (0xcc1 bytes) 000195[0] 1766413240.150003 mapread 0x35bee8 thru 0x35dcab (0x1dc4 bytes) 000198[1] 1766413240.296753 mapwrite 0x4302de thru 0x431157 (0xe7a bytes) 000199[0] 1766413240.481805 mapwrite 0x2e7039 thru 0x2e7629 (0x5f1 bytes) 000200[0] 1766413240.684056 mapread 0x50dd94 thru 0x50de3a (0xa7 bytes) 000201[1] 1766413240.756081 trunc from 0x662a48 to 0x8a5632 (0x242beb bytes) 000203[1] 1766413240.862312 mapwrite 0x362527 thru 0x363b27 (0x1601 bytes) 000204[0] 1766413240.945000 write 0x3a9154 thru 0x3a99e8 (0x895 bytes) 000213[0] 1766413241.157053 mapwrite 0x872bdb thru 0x872d42 (0x168 bytes) 000215[1] 1766413241.230954 read 0x259b9b thru 0x25acb0 (0x1116 bytes) 000216[1] 1766413241.263824 mapwrite 0x477562 thru 0x4794d6 (0x1f75 bytes) 000219[1] 1766413241.431396 mapread 0x6416f2 thru 0x641dd0 (0x6df bytes) 000221[0] 1766413241.500021 mapread 0x638837 thru 0x638989 (0x153 bytes) 000225[1] 1766413241.547483 write 0x35e3a2 thru 0x35ff0c (0x1b6b bytes) 000228[1] 1766413241.583888 mapread 0x204b0e thru 0x2060e3 (0x15d6 bytes) 000229[0] 1766413241.625777 read 0x461b05 thru 0x461c86 (0x182 bytes) 000232[0] 1766413241.732530 write 0x7ee518 thru 0x7ef99d (0x1486 bytes) 000233[1] 1766413241.783908 mapwrite 0x6fd2ef thru 0x6fd8eb (0x5fd bytes) 000235[0] 1766413241.904030 mapwrite 0x9e6623 thru 0x9e810e (0x1aec bytes) 000236[0] 1766413241.997191 mapread 0x460ff0 thru 0x462c76 (0x1c87 bytes) 000237[0] 1766413242.013233 write 0x13968f thru 0x13ae80 (0x17f2 bytes) 000239[1] 1766413242.022249 trunc from 0x9e810f to 0x5d04f6 (0x417c18 bytes) 000241[1] 1766413242.098399 trunc from 0x5d04f7 to 0x6d539e (0x104ea8 bytes) 000242[0] 1766413242.151569 write 0x736056 thru 0x736be6 (0xb91 bytes) 000243[0] 1766413242.186310 trunc from 0x736be7 to 0x339e73 (0x3fcd73 bytes) 000244[1] 1766413242.233605 mapread 0x0edce8 thru 0x0efbc4 (0x1edd bytes) 000246[0] 1766413242.296733 read 0x05ec05 thru 0x05faa5 (0xea1 bytes) 000248[0] 1766413242.427259 mapread 0x1573d6 thru 0x158242 (0xe6d bytes) 000249[0] 1766413242.428012 mapread 0x08ed8c thru 0x090452 (0x16c7 bytes) 000250[1] 1766413242.428575 write 0x4203b6 thru 0x4219ed (0x1638 bytes) 000251[0] 1766413242.444521 write 0x3dbb1e thru 0x3dc669 (0xb4c bytes) 000255[1] 1766413242.549265 write 0x241a98 thru 0x242048 (0x5b1 bytes) 000256[1] 1766413242.586742 trunc from 0x4219ee to 0x6a2f20 (0x281533 bytes) 000259[1] 1766413242.609311 mapread 0x627e02 thru 0x628060 (0x25f bytes) 000260[0] 1766413242.640583 read 0x1f575b thru 0x1f6c2e (0x14d4 bytes) truncating to largest ever: 0x9001b2 000261[0] 1766413242.706246 trunc from 0x6a2f21 to 0x9001b1 (0x25d291 bytes) 000263[1] 1766413242.791534 mapread 0x5a6b38 thru 0x5a8329 (0x17f2 bytes) 000264[0] 1766413242.841718 mapread 0x6ea8b7 thru 0x6ec11b (0x1865 bytes) 000265[0] 1766413242.857865 mapread 0x809ae8 thru 0x80afdd (0x14f6 bytes) 000266[0] 1766413242.867282 trunc from 0x9001b2 to 0x66e302 (0x291eaf bytes) 000267[1] 1766413242.901215 mapwrite 0x611297 thru 0x611ef2 (0xc5c bytes) truncating to largest ever: 0x9eaf58 000268[0] 1766413242.983253 trunc from 0x66e303 to 0x9eaf57 (0x37cc55 bytes) 000269[1] 1766413243.047301 mapwrite 0x76d8d2 thru 0x76e38d (0xabc bytes) 000271[1] 1766413243.248244 mapread 0x78ce4d thru 0x78d4ab (0x65f bytes) 000272[1] 1766413243.310728 mapwrite 0x9673bb thru 0x967f8e (0xbd4 bytes) 000276[0] 1766413243.441054 mapwrite 0x630ce9 thru 0x6315ab (0x8c3 bytes) 000277[0] 1766413243.539752 read 0x8ee25b thru 0x8efa02 (0x17a8 bytes) 000278[0] 1766413243.583165 write 0x43aa90 thru 0x43c3ea (0x195b bytes) 000281[0] 1766413243.605691 read 0x6e0de5 thru 0x6e0e1c (0x38 bytes) 000282[0] 1766413243.611361 mapread 0x67f5b6 thru 0x68053e (0xf89 bytes) 000284[1] 1766413243.639633 write 0x792792 thru 0x792a2a (0x299 bytes) 000285[1] 1766413243.663436 trunc from 0x9eaf58 to 0x521833 (0x4c9724 bytes) 000287[1] 1766413243.728572 mapwrite 0x3abab7 thru 0x3acdb3 (0x12fd bytes) 000288[1] 1766413243.789602 read 0x121978 thru 0x122fb0 (0x1639 bytes) 000292[1] 1766413243.847549 trunc from 0x521834 to 0x7aa008 (0x2887d5 bytes) 000293[1] 1766413243.910424 mapread 0x64a6ea thru 0x64afc5 (0x8dc bytes) 000295[0] 1766413243.958975 mapwrite 0x3b06d7 thru 0x3b0924 (0x24e bytes) 000298[1] 1766413244.059315 trunc from 0x7aa009 to 0x4d61a2 (0x2d3e66 bytes) 000299[0] 1766413244.090987 trunc from 0x4d61a3 to 0x4da098 (0x3ef6 bytes) 000301[0] 1766413244.135829 read 0x23502f thru 0x235479 (0x44b bytes) 000302[0] 1766413244.160206 mapwrite 0x80f23a thru 0x810768 (0x152f bytes) 000303[0] 1766413244.238532 trunc from 0x810769 to 0x32eec4 (0x4e18a4 bytes) 000304[1] 1766413244.279525 mapwrite 0x78b995 thru 0x78cc91 (0x12fd bytes) 000306[1] 1766413244.402235 read 0x5a1e18 thru 0x5a310e (0x12f7 bytes) 000310[1] 1766413244.469813 mapwrite 0x7f02aa thru 0x7f176b (0x14c2 bytes) 000312[0] 1766413244.632957 mapwrite 0x8b96c2 thru 0x8bb511 (0x1e50 bytes) 000313[1] 1766413244.763102 mapwrite 0x95ae7f thru 0x95c433 (0x15b5 bytes) 000316[1] 1766413244.879073 mapread 0x229d91 thru 0x22a91a (0xb8a bytes) 000318[0] 1766413244.908650 mapread 0x8a7eaf thru 0x8a9987 (0x1ad9 bytes) 000323[0] 1766413244.993701 read 0x3bd117 thru 0x3bf0e7 (0x1fd1 bytes) 000325[1] 1766413245.023986 mapwrite 0x927ca5 thru 0x929050 (0x13ac bytes) 000326[1] 1766413245.088450 write 0x0fe476 thru 0x0fffe2 (0x1b6d bytes) 000333[1] 1766413245.161271 mapread 0x1a7231 thru 0x1a7f5a (0xd2a bytes) 000334[1] 1766413245.170824 read 0x756905 thru 0x7573a6 (0xaa2 bytes) 000335[0] 1766413245.175794 mapwrite 0x113f7b thru 0x1151d6 (0x125c bytes) 000337[1] 1766413245.311690 mapwrite 0x9efbd4 thru 0x9f032e (0x75b bytes) 000340[1] 1766413245.507826 mapread 0x6cc344 thru 0x6cce5e (0xb1b bytes) 000341[0] 1766413245.536838 trunc from 0x9f032f to 0x9ce8d1 (0x21a5d bytes) 000344[0] 1766413245.607262 write 0x3646bd thru 0x365dab (0x16ef bytes) 000348[0] 1766413245.663229 mapwrite 0x102e54 thru 0x1048c7 (0x1a74 bytes) 000350[0] 1766413245.751873 write 0x757360 thru 0x7574c2 (0x163 bytes) 000354[0] 1766413245.800488 mapread 0x8228f8 thru 0x823874 (0xf7d bytes) 000357[0] 1766413245.851661 mapwrite 0x41f98f thru 0x420d7d (0x13ef bytes) 000358[0] 1766413245.896374 trunc from 0x9ce8d2 to 0x90c65a (0xc2277 bytes) 000362[0] 1766413245.977794 read 0x714ccd thru 0x715b2a (0xe5e bytes) 000366[0] 1766413245.999692 read 0x8e9909 thru 0x8e9d6e (0x466 bytes) 000370[1] 1766413246.110291 read 0x855dea thru 0x856193 (0x3aa bytes) 000372[0] 1766413246.143576 mapread 0x752603 thru 0x7528d8 (0x2d6 bytes) 000373[1] 1766413246.148160 write 0x1dec6d thru 0x1df717 (0xaab bytes) 000375[1] 1766413246.189673 mapwrite 0x088080 thru 0x0884ae (0x42f bytes) 000376[0] 1766413246.267717 mapread 0x12cfec thru 0x12ed61 (0x1d76 bytes) 000380[0] 1766413246.360888 write 0x4a13f5 thru 0x4a2c28 (0x1834 bytes) 000383[1] 1766413246.408775 read 0x3502fc thru 0x351424 (0x1129 bytes) 000386[1] 1766413246.454794 mapwrite 0x54e742 thru 0x5504ed (0x1dac bytes) 000390[1] 1766413246.587043 mapread 0x52236f thru 0x523f2b (0x1bbd bytes) 000399[1] 1766413246.696340 write 0x445eb0 thru 0x44796a (0x1abb bytes) 000403[1] 1766413246.769694 write 0x6ec79d thru 0x6ed0fe (0x962 bytes) 000404[0] 1766413246.810667 mapwrite 0x207a7d thru 0x209811 (0x1d95 bytes) 000406[0] 1766413246.944496 trunc from 0x90c65b to 0x167314 (0x7a5346 bytes) 000408[1] 1766413247.039851 write 0x264fff thru 0x265a0a (0xa0c bytes) 000410[0] 1766413247.103281 read 0x0f4a6f thru 0x0f6046 (0x15d8 bytes) 000418[1] 1766413247.243871 write 0x5a9fe9 thru 0x5aa66e (0x686 bytes) 000419[0] 1766413247.270491 mapread 0x44bb46 thru 0x44ce3d (0x12f8 bytes) 000421[1] 1766413247.315923 trunc from 0x5aa66f to 0x05ed23 (0x54b94b bytes) 000427[1] 1766413247.429786 write 0x03c840 thru 0x03dc1b (0x13dc bytes) 000430[1] 1766413247.509089 write 0x72772c thru 0x727781 (0x56 bytes) 000433[0] 1766413247.574015 trunc from 0x727782 to 0x418b20 (0x30ec61 bytes) 000437[0] 1766413247.703275 read 0x15a4f0 thru 0x15b706 (0x1217 bytes) 000438[0] 1766413247.712840 trunc from 0x418b21 to 0x6f0adc (0x2d7fbc bytes) 000442[0] 1766413247.811064 read 0x1dd593 thru 0x1df23e (0x1cac bytes) 000443[0] 1766413247.822316 trunc from 0x6f0add to 0x4b1b64 (0x23ef78 bytes) 000447[1] 1766413247.905859 mapwrite 0x1f945d thru 0x1f94ed (0x91 bytes) 000452[0] 1766413248.033187 read 0x2e9f91 thru 0x2eadb3 (0xe23 bytes) 000453[0] 1766413248.057752 mapwrite 0x45bade thru 0x45c1af (0x6d2 bytes) 000455[0] 1766413248.147421 mapwrite 0x6e07b2 thru 0x6e1a2c (0x127b bytes) 000456[0] 1766413248.321158 mapwrite 0x198df0 thru 0x19a8a5 (0x1ab6 bytes) 000458[0] 1766413248.462549 read 0x580c05 thru 0x581e54 (0x1250 bytes) 000460[0] 1766413248.472586 read 0x65ee4f thru 0x65f609 (0x7bb bytes) 000461[1] 1766413248.504801 trunc from 0x6e1a2d to 0x5cbd0b (0x115d21 bytes) 000462[1] 1766413248.559862 mapread 0x448bd6 thru 0x44a851 (0x1c7c bytes) 000463[1] 1766413248.625128 mapwrite 0x1a51e1 thru 0x1a66ed (0x150d bytes) 000468[1] 1766413248.767048 read 0x2f48b7 thru 0x2f4d68 (0x4b2 bytes) 000469[0] 1766413248.779628 write 0x5b66c3 thru 0x5b715f (0xa9d bytes) 000470[1] 1766413248.826645 read 0x23a45f thru 0x23bdff (0x19a1 bytes) 000471[1] 1766413248.851206 trunc from 0x5cbd0c to 0x08feaf (0x53be5c bytes) 000473[1] 1766413248.925175 mapread 0x039341 thru 0x03a73d (0x13fd bytes) 000476[1] 1766413248.995596 trunc from 0x08feb0 to 0x3e9794 (0x3598e5 bytes) 000480[1] 1766413249.090813 mapread 0x060377 thru 0x060aae (0x738 bytes) 000485[1] 1766413249.156024 write 0x99ad8d thru 0x99b09d (0x311 bytes) 000487[1] 1766413249.178545 mapwrite 0x3a75f1 thru 0x3a7b70 (0x580 bytes) 000488[0] 1766413249.241585 mapread 0x52c2f6 thru 0x52d2a2 (0xfad bytes) 000490[0] 1766413249.358707 mapread 0x00add8 thru 0x00b067 (0x290 bytes) 000493[1] 1766413249.427963 mapread 0x95d2f9 thru 0x95eb12 (0x181a bytes) 000495[1] 1766413249.439611 write 0x1f9884 thru 0x1fa340 (0xabd bytes) 000496[1] 1766413249.457731 read 0x51481f thru 0x51529c (0xa7e bytes) 000497[0] 1766413249.472981 mapwrite 0x314e05 thru 0x316b4d (0x1d49 bytes) 000498[0] 1766413249.577506 mapread 0x95f3ca thru 0x95f8c5 (0x4fc bytes) All operations completed A-OK! PASS 16b (26s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:20:54 (1766413254) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:20:58 (1766413258) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.346018 s, 24.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.242191 s, 34.6 MB/s ldlm.namespaces.MGC192.168.202.128@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c1106845000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c1119f79000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c1106845000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c1106845000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.3278 s, 11.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.97827 s, 35.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.37915 s, 44.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.26574 s, 16.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.68619 s, 12.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.29647 s, 31.8 MB/s PASS 16d (44s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:21:42 (1766413302) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.182565 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15446 s, 6.8 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:21:48 (1766413308) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:22:15 (1766413335) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 09:22:42 (1766413362) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 09:22:48 (1766413368) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 09:22:55 (1766413375) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05839 s, 12.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5087 s, 17.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48836 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49852 s, 17.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.92251 s, 13.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87021 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.27374 s, 11.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20942 s, 21.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55237 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.82026 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.93005 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.35711 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53215 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95378 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.21695 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.94132 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18211 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.93256 s, 6.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.98958 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (35s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:23:30 (1766413410) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Dec 22 09:24:12 EST 2025 + fsx 2 done: rc=0 Mon Dec 22 09:24:12 EST 2025 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Mon Dec 22 09:24:12 EST 2025 + fsx 4 done: rc=0 Mon Dec 22 09:24:12 EST 2025 + fsx 5 done: rc=0 Mon Dec 22 09:24:12 EST 2025 PASS 16k (50s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:24:20 (1766413460) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 09:24:30 (1766413470) mmap test1: basic mmap operation (PASS, 0.092362s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10759s) mmap test3: concurrent mmap ops on two nodes (PASS, 20.226s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5447s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.25311s) mmap test6: check mmap write/read content on two nodes (PASS, 0.28323s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2299s) mmap test11: multi-thread shared mmap (PASS, 1.4779s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0999s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:25:07 (1766413507) oleg228-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:25:11 (1766413511) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:25:17 (1766413517) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 09:25:23 (1766413523) now is 1766413525 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 new atime is 1766413586 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:26:31 (1766413591) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:26:36 (1766413596) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:26:42 (1766413602) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:26:48 (1766413608) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 09:26:51 (1766413611) utime: good mknod times 1766413611 <= 1766413612 <= 1766413612 for /mnt/lustre/f26a utime: good mknod times 1766413611 <= 1766413612 <= 1766413612 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 09:26:57 (1766413617) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:27:06 (1766413626) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 09:27:07.000000000 -0500 Birth: 2025-12-22 09:27:07.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 09:27:07.000000000 -0500 Birth: 2025-12-22 09:27:07.000000000 -0500 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:27:14 (1766413634) 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:27:21 (1766413641) fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:27:27 (1766413647) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.128@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 09:27:45 (1766413665) -rw-r--r-- 1 root root 0 Dec 22 09:27 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (8s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:27:52 (1766413672) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42887 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:27:58 (1766413678) PASS 31t (9s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:28:08 (1766413688) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:28:12 (1766413692) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:28:16 (1766413696) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:28:19 (1766413699) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 09:28:22 (1766413702) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid in IDLE state after 0 sec oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid in FULL state after 0 sec oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid in IDLE state after 0 sec oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid in FULL state after 0 sec oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1106845000.ost_server_uuid in IDLE state after 0 sec oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1106845000.ost_server_uuid in FULL state after 0 sec PASS 34 (122s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:30:25 (1766413825) fail_loc=0x80000317 Race attempt 0 Wait for 48941 48979 for 60 sec... - open/close 779 (time 1766413841.48 total 10.00 last 77.90) - open/close 1487 (time 1766413851.49 total 20.00 last 70.77) - open/close 2287 (time 1766413861.50 total 30.01 last 79.94) - open/close 3089 (time 1766413871.50 total 40.02 last 80.17) - open/close 3905 (time 1766413881.51 total 50.02 last 81.54) total: 4000 open/close in 51.10 seconds: 78.28 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.128@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.128@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:31:36 (1766413896) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39377 s, 37.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.62192 s, 32.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.889319 s, 59.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.943057 s, 55.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.955728 s, 54.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.958166 s, 54.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51249 s, 34.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.17068 s, 44.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08405 s, 48.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.993233 s, 52.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7490560) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0414 s, 50.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7445504) after(7496704) PASS 36 (294s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 09:36:30 (1766414190) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 - create 1481 (time 1766414203.75 total 10.01 last 148.02) - create 3070 (time 1766414213.75 total 20.01 last 158.87) - create 4767 (time 1766414223.76 total 30.01 last 169.62) - create 6303 (time 1766414233.76 total 40.01 last 153.58) - create 7922 (time 1766414243.76 total 50.02 last 161.83) - create 9523 (time 1766414253.77 total 60.02 last 160.05) total: 10000 create in 63.03 seconds: 158.66 ops/second PASS 37 (72s) == sanityn test 39a: file mtime does not change after rename ========================================================== 09:37:42 (1766414262) repeat after cancel_lru_locks PASS 39a (6s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 09:37:48 (1766414268) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 09:37:55 (1766414275) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 09:38:02 (1766414282) fail_loc=0x411 fail_loc=0 PASS 39d (5s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 09:38:09 (1766414289) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (15s) == sanityn test 40c: pdirops: link and others ======================================================================== 09:38:24 (1766414304) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (15s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 09:38:39 (1766414319) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 09:38:54 (1766414334) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (13s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 09:39:08 (1766414348) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (10s) == sanityn test 41b: pdirops: create vs create ======================================================================== 09:39:18 (1766414358) open: File exists Conflict PASS 41b (11s) == sanityn test 41c: pdirops: create vs link ======================================================================== 09:39:29 (1766414369) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (11s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 09:39:40 (1766414380) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 09:39:52 (1766414392) Conflict PASS 41e (11s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 09:40:04 (1766414404) Conflict PASS 41f (11s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 09:40:15 (1766414415) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 09:40:26 (1766414426) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 09:40:38 (1766414438) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (962s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 09:56:40 (1766415400) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (10s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 09:56:50 (1766415410) open: File exists Conflict PASS 42b (10s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 09:57:00 (1766415420) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (10s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 09:57:10 (1766415430) Conflict PASS 42d (9s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 09:57:19 (1766415439) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (10s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 09:57:29 (1766415449) Conflict PASS 42f (9s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 09:57:38 (1766415458) Conflict PASS 42g (10s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 09:57:48 (1766415468) Conflict PASS 42h (9s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 09:57:57 (1766415477) PASS 43a (47s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 09:58:44 (1766415524) Conflict PASS 43b (9s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 09:58:53 (1766415533) Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 09:59:00 (1766415540) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (9s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 09:59:09 (1766415549) Conflict PASS 43e (8s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 09:59:17 (1766415557) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (8s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 09:59:25 (1766415565) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 09:59:32 (1766415572) Conflict PASS 43h (8s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 09:59:40 (1766415580) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 09:59:42 (1766415582) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (79s) == sanityn test 43k: unlink vs create ==================== 10:01:01 (1766415661) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (593s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:10:54 (1766416254) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Dec 22 10:10:58 EST 2025 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:11:00 (1766416260) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:11:07 (1766416267) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:11:13 (1766416273) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:11:19 (1766416279) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:11:25 (1766416285) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:11:31 (1766416291) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:11:38 (1766416298) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:11:44 (1766416304) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:11:46 (1766416306) PASS 45a (48s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:12:34 (1766416354) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:12:40 (1766416360) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:12:46 (1766416366) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:12:52 (1766416372) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:12:58 (1766416378) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:13:04 (1766416384) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:13:10 (1766416390) Conflict PASS 45h (6s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:13:16 (1766416396) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 10:13:17 (1766416397) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (463s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 10:21:00 (1766416860) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 10:21:06 (1766416866) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 10:21:11 (1766416871) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 10:21:16 (1766416876) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 10:21:22 (1766416882) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 10:21:27 (1766416887) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 10:21:32 (1766416892) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 10:21:38 (1766416898) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 10:21:43 (1766416903) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 10:21:44 (1766416904) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 10:21:46 (1766416906) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 10:21:47 (1766416907) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 10:21:48 (1766416908) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 10:21:49 (1766416909) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 10:21:50 (1766416910) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 10:21:51 (1766416911) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 10:21:53 (1766416913) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0060821 s, 1.7 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 10:22:00 (1766416920) 0+1 records in 0+1 records out 158 bytes copied, 0.00500575 s, 31.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216020) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 10:22:05 (1766416925) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00362008 s, 283 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197296 s, 53.1 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 10:22:19 (1766416939) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00606778 s, 169 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 10:22:26 (1766416946) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163307 s, 64.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 10:22:30 (1766416950) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 10:22:34 (1766416954) llite.lustre-ffff9c1106845000.xattr_cache=1 llite.lustre-ffff9c1119f79000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:34.000000000 -0500 Modify: 2025-12-22 10:22:34.000000000 -0500 Change: 2025-12-22 10:22:34.000000000 -0500 Birth: 2025-12-22 10:22:35.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:40.000000000 -0500 Modify: 2025-12-22 10:22:40.000000000 -0500 Change: 2025-12-22 10:22:40.000000000 -0500 Birth: 2025-12-22 10:22:40.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:46.000000000 -0500 Modify: 2025-12-22 10:22:46.000000000 -0500 Change: 2025-12-22 10:22:46.000000000 -0500 Birth: 2025-12-22 10:22:46.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:46.000000000 -0500 Modify: 2025-12-22 10:22:46.000000000 -0500 Change: 2025-12-22 10:22:46.000000000 -0500 Birth: 2025-12-22 10:22:46.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:51.000000000 -0500 Modify: 2025-12-22 10:22:51.000000000 -0500 Change: 2025-12-22 10:22:51.000000000 -0500 Birth: 2025-12-22 10:22:51.000000000 -0500 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:22:51.000000000 -0500 Modify: 2025-12-22 10:22:51.000000000 -0500 Change: 2025-12-22 10:22:51.000000000 -0500 Birth: 2025-12-22 10:22:51.000000000 -0500 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 10:22:58 (1766416978) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 10:23:06 (1766416986) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 10:23:14 (1766416994) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 10:23:27 (1766417007) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 10:23:37 (1766417017) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 10:23:38 (1766417018) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 10:23:52 (1766417032) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 10:24:06 (1766417046) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1766417047 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.6497 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (41s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 10:24:47 (1766417087) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1766417088 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.48783 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (39s) == sanityn test 60: Verify data_version behaviour ======== 10:25:26 (1766417126) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 10:25:28 (1766417128) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 10:25:31 (1766417131) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 10:25:34 (1766417134) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0033445 s, 19.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 10:25:35 (1766417135) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00406452 s, 16.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 10:25:36 (1766417136) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 10:25:37 (1766417137) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00394259 s, 16.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 10:25:39 (1766417139) llite.lustre-ffff9c1106845000.xattr_cache=1 llite.lustre-ffff9c1119f79000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 10:25:41 (1766417141) llite.lustre-ffff9c1106845000.xattr_cache=1 llite.lustre-ffff9c1119f79000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 10:25:43 (1766417143) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00256041 s, 400 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00283837 s, 361 kB/s 228785: get lock2 228787: get lock1 228785: done 228785: get lock1 228787: done 228787: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 228785: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 228787: cannot get lock2: Resource deadlock avoided 228787: exit rc=0 228785: done 228785: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 10:25:49 (1766417149) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0258099 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989985 s, 106 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229368 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 10:25:56 (1766417156) opening files: ........ closing files: ........ PASS 76 (18s) == sanityn test 77a: check FIFO NRS policy =============== 10:26:15 (1766417175) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123028 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361597 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060181 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899832 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901299 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872564 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722586 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972399 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074591 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896487 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132294 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139748 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15261 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170562 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14501 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160685 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18375 s, 5.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155314 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194438 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174718 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947007 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654611 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176381 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229079 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143122 s, 73.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298957 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172033 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158532 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118472 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673955 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107407 s, 97.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167423 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138527 s, 75.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 10:26:17 (1766417177) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143245 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152195 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262544 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778809 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672882 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630111 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855546 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906593 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785638 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785235 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867709 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830728 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971079 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094591 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835599 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103735 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891371 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110138 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189323 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00688499 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155907 s, 67.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00898244 s, 117 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144051 s, 72.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103746 s, 101 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508363 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182782 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00825147 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013833 s, 75.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167386 s, 62.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112444 s, 93.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583234 s, 180 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179713 s, 58.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101634 s, 103 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127835 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238678 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440265 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337162 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415102 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04756 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103368 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100883 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104822 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101324 s, 10.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101721 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11741 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116789 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126198 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105501 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113608 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126233 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166122 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114981 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018327 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179583 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015706 s, 66.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171965 s, 61.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122262 s, 85.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260357 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02338 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102909 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187785 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267427 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723231 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112068 s, 93.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856608 s, 122 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208505 s, 50.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 10:26:21 (1766417181) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163798 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403114 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653206 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615766 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779143 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834309 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907817 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640907 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709329 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900505 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932864 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104638 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101781 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930498 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10344 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957185 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115441 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149832 s, 70.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208462 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121053 s, 86.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132707 s, 79.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157856 s, 66.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190019 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015728 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249384 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261016 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192639 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025232 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277192 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548902 s, 191 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216909 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308848 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279333 s, 37.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119073 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226228 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322128 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776763 s, 13.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111597 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878178 s, 11.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105258 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100369 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109678 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11662 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106333 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901629 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121189 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13481 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124563 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13063 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102913 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090943 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150677 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173502 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204003 s, 51.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256543 s, 40.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187429 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201848 s, 51.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132607 s, 79.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022293 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228703 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238694 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091321 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149969 s, 69.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146045 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708713 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293433 s, 35.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122397 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345287 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661028 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673156 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640259 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842889 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825282 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696805 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632383 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10692 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992662 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950942 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886947 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10947 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100828 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088744 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109933 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139043 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147501 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165253 s, 63.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179969 s, 58.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215056 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020717 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010476 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109065 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136555 s, 76.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672946 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240701 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059178 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230282 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020026 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143397 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622811 s, 168 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 10:26:26 (1766417186) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159181 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681012 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783386 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654415 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067836 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731987 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954833 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940524 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940634 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837877 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086217 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105179 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118559 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127382 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130791 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140425 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141187 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156478 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201079 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182401 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210751 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133509 s, 78.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133669 s, 78.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119108 s, 88.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249448 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410217 s, 256 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246028 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367739 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253054 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641395 s, 163 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268962 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024712 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279832 s, 37.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=5 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12224 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373437 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435034 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563644 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045215 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058046 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449247 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699359 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525817 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666246 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106527 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120011 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117753 s, 8.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110136 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124753 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107915 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147554 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159332 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136912 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176478 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029731 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019696 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156559 s, 67.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219667 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163133 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620973 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910222 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242674 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236103 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028692 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448717 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382993 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507726 s, 207 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145122 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326473 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511134 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576989 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640751 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422265 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642966 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657681 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577068 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486064 s, 21.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839409 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920914 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109697 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864242 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10071 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104981 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905281 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167167 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013761 s, 76.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141057 s, 74.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211647 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211246 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801909 s, 131 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00838113 s, 125 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155166 s, 67.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188464 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369633 s, 284 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133241 s, 78.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642791 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259161 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319966 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184865 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121201 s, 86.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 10:26:31 (1766417191) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.28@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17914 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226062 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036242 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376889 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532389 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698829 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800943 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811351 s, 12.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828973 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898772 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112449 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107614 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10712 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103483 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126652 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945093 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119492 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787673 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076423 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777601 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794859 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786444 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770364 s, 13.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751845 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804394 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752525 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796015 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832984 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847347 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794616 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817329 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080056 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083155 s, 12.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143298 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264996 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551972 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481753 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699452 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690023 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771716 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661348 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705189 s, 14.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834078 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932342 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994783 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102208 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112033 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116493 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121171 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126255 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755127 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08694 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092966 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859765 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854222 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078008 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865558 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871879 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840021 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831901 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808638 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904936 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982544 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838515 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862308 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08926 s, 11.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117778 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231461 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529692 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474596 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593128 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055134 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547052 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713804 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511155 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489132 s, 21.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071067 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872845 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103928 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992069 s, 10.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112204 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117417 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141143 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183162 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215511 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197627 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014267 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239165 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495062 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205155 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183613 s, 57.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208788 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376891 s, 278 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239145 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306535 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278231 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989649 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723194 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443203 s, 237 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147245 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261988 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656478 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051216 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533751 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687521 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726665 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646659 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692483 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779458 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12926 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137435 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887136 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100704 s, 10.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148062 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104572 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161848 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167342 s, 62.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00894554 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106666 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170894 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222128 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226417 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268903 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00933063 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451879 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720185 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124899 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416098 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112773 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108107 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262694 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246489 s, 42.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:26:39 (1766417199) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147258 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311312 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255966 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846766 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799937 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898227 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720127 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710096 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10659 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102144 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863198 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969373 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828448 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101795 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988893 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098479 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110108 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160767 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159307 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161497 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141213 s, 74.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154099 s, 68.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155801 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198233 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233776 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176622 s, 59.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192217 s, 54.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857338 s, 122 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971081 s, 108 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790622 s, 133 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432148 s, 243 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491451 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900916 s, 116 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161005 s, 104 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295942 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404624 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586169 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750262 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664259 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801121 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870355 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101919 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961421 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900368 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101188 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892174 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108686 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105159 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108581 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104253 s, 10.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160985 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163357 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183455 s, 57.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163161 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168651 s, 62.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122838 s, 85.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144565 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129441 s, 81.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196416 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121013 s, 86.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196859 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205761 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00530282 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420768 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00524068 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362116 s, 290 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155491 s, 108 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228056 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345749 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105374 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10854 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102022 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11973 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117719 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114253 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119758 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148628 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135054 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131305 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129999 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14494 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149578 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152745 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122635 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131042 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150742 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185368 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175156 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264699 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00649349 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241912 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255071 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736815 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020784 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654759 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339333 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447586 s, 23.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00341977 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275146 s, 381 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136586 s, 123 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368479 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327053 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312806 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327244 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449436 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614395 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068999 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645547 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914532 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103181 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1119f79000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928614 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558928 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927823 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117905 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108547 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100048 s, 10.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159652 s, 65.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138097 s, 75.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232881 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193968 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232267 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109924 s, 95.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1119f79000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042996 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103219 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150108 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00666817 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225941 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00501599 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160834 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260287 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209176 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104446 s, 100 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 10:26:46 (1766417206) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:26:50 (1766417210) oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg228-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg228-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg228-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg228-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 10:26:54 (1766417214) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:27:01 (1766417221) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125332 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240339 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634678 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085677 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936383 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921318 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091331 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797134 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788502 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922329 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936471 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947638 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899122 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834264 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899192 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891049 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149174 s, 70.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139199 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164494 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152529 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209263 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215403 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728574 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225642 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160273 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103304 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198394 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221185 s, 47.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00345995 s, 303 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453503 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499357 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00192862 s, 544 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99199 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.429 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:27:47 (1766417267) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455197 s, 36.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400575 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454519 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837149 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899121 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801189 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936797 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925435 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745299 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128048 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123148 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142021 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130213 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131621 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117353 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136138 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143381 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165113 s, 63.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118651 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173439 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129036 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147787 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146981 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140582 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160777 s, 65.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011918 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698478 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00922187 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239891 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371541 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384775 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527572 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569834 s, 184 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8315 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9579 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22773 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554709 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642631 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662707 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784412 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628692 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575784 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540876 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521851 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160332 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16044 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142072 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158417 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163488 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150972 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186959 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18581 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019844 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199927 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225424 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242763 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023166 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214089 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285343 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206754 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189525 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167583 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430634 s, 243 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299353 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402873 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039749 s, 264 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104799 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331689 s, 31.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8294 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9552 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (138s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:30:05 (1766417405) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23101 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691922 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650089 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961421 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087674 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809132 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101856 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108852 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937259 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156218 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156269 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140433 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172125 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144309 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14367 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177336 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163979 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017108 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186881 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174037 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210319 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173463 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179271 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226294 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167388 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014687 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635979 s, 165 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193565 s, 54.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138987 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00336741 s, 311 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112406 s, 93.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942477 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444638 s, 236 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8353 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9516 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22192 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712507 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651574 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716404 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882454 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801276 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093847 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734019 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839569 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157323 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16196 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154984 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157612 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170161 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182624 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177429 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159571 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156142 s, 67.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200665 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181513 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182826 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218308 s, 48.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157074 s, 66.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791947 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522076 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253795 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227133 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242205 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403445 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987758 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127949 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159973 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241915 s, 43.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8321 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9565 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (137s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:32:22 (1766417542) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.259093 s, 64.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437688 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519726 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473631 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066185 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467088 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477162 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655074 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519003 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711311 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098117 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966934 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11364 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12459 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100981 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120161 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128488 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127407 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184945 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176652 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178602 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175153 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015458 s, 67.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446933 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240959 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423217 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246447 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560592 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862187 s, 122 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477037 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104796 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543018 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311735 s, 33.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94965 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73643 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.28@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.28@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257974 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348046 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425298 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410507 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128336 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125306 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139914 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136854 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130805 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149405 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153412 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147474 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134968 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148716 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145979 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256584 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25167 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019503 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182346 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013793 s, 76.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175263 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158172 s, 66.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294786 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201314 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257837 s, 40.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154491 s, 67.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343855 s, 305 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286964 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024804 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886965 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126789 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450448 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113072 s, 92.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98903 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73196 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.28@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150125 s, 112 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237107 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427447 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481288 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432182 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378299 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568321 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667293 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926194 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979625 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109533 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983772 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677262 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952515 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104113 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105034 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113938 s, 9.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123312 s, 85.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142323 s, 73.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219694 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222219 s, 47.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225313 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550536 s, 190 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189625 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198122 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139239 s, 75.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248883 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167523 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414534 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791694 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153253 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903983 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548734 s, 191 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97789 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.8811 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159368 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269105 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494468 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618926 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585791 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712496 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480836 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563993 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565605 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810915 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648948 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106982 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102414 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141141 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148972 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250034 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258078 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154265 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274715 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02115 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277995 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122566 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287903 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205527 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324366 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276028 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239127 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338877 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287803 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202437 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159997 s, 65.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119812 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260535 s, 40.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77817 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7339 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91341 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73193 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.856296 s, 19.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469616 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350148 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416357 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529422 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508257 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045715 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516655 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585466 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155875 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180667 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177276 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162754 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204532 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191459 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200101 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210481 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173146 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132138 s, 79.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140296 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134648 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138958 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186699 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00714475 s, 147 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214904 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247623 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542386 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200264 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673197 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160016 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719342 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117887 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00187175 s, 560 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7604 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.431 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139529 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218156 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483749 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071349 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768208 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10133 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936936 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942823 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961001 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113345 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105191 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951644 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106382 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930508 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898418 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103079 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111249 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176993 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170245 s, 61.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156976 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174621 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183651 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223296 s, 47.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171264 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171703 s, 61.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135505 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962821 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436997 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121848 s, 86.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292777 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00715657 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394032 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755758 s, 139 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93865 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73089 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132522 s, 127 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284075 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594501 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575593 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054298 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722908 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066119 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562728 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704896 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661872 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138521 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132168 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136298 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14964 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136149 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168578 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170006 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135401 s, 77.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149803 s, 70.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176815 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190317 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152599 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185399 s, 56.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0029424 s, 356 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173177 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206786 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473464 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017075 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231228 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113803 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354396 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375982 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460482 s, 228 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87144 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73643 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76795 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7294 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (365s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 10:38:27 (1766417907) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247659 s, 67.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239771 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330416 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077507 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653311 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980663 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737282 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108692 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116495 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990221 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11198 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105991 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109776 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115065 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107507 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107115 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108174 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142227 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213434 s, 49.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155136 s, 67.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101095 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162514 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222137 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201871 s, 51.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282398 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024046 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021699 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223584 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275462 s, 38.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224058 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00526945 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262041 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116377 s, 90.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98289 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73286 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 10:39:04 (1766417944) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.28@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.28@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140432 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267284 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435515 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609304 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903118 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625601 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578656 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836142 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959377 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837899 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102203 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161204 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160185 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20283 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204863 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24968 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26832 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192625 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182812 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142803 s, 73.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177873 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144319 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436398 s, 240 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622911 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405327 s, 259 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251503 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021372 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153954 s, 68.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231315 s, 45.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159984 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416351 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628835 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225367 s, 465 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01833 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73233 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 10:39:45 (1766417985) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.28@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.26959 s, 62.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385004 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839207 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810169 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973997 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996803 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920944 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10919 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104508 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943974 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785342 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887678 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860758 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988845 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976417 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107032 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985234 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173147 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186671 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016343 s, 64.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235154 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243561 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306425 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116569 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202968 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144127 s, 72.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785185 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351313 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301439 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168443 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128352 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598076 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189955 s, 55.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98011 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96998 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 10:40:21 (1766418021) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443571 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249453 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448939 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617181 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527104 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610791 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650842 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481237 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648675 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874648 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101918 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879358 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102244 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315298 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315319 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354868 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350235 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136316 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165367 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194012 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157418 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167661 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206325 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424799 s, 247 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139142 s, 75.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860433 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00838777 s, 125 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275299 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240671 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391207 s, 268 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115027 s, 91.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268707 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014145 s, 74.1 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg228-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg228-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79573 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73672 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92181 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73488 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 10:41:38 (1766418098) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.152067 s, 110 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025375 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509724 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540868 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652897 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460065 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735136 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635589 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701991 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739938 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881385 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884249 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102984 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989642 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878775 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114642 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115171 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010918 s, 96.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146693 s, 71.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161619 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194306 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198741 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00926491 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137684 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200278 s, 52.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191942 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420941 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250588 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328836 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418952 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412799 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486708 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00270307 s, 388 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6628 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9381 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (65s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 10:42:43 (1766418163) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252115 s, 66.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248679 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528514 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763337 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829356 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078496 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831075 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108112 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103179 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921278 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896098 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10506 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111966 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889955 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777282 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110653 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093505 s, 11.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140511 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157414 s, 66.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227228 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231127 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231181 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241881 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154272 s, 68.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105158 s, 99.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817001 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00964813 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162817 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272968 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137141 s, 76.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243571 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143061 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663225 s, 158 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98815 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73657 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg228-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164625 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231194 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382495 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434732 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545743 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720493 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738631 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705874 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745564 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059595 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812256 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10104 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104983 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128562 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150635 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184793 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180535 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184287 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186176 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186283 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188372 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171764 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183801 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393644 s, 266 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00461209 s, 227 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539639 s, 194 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171627 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266756 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198895 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279546 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109081 s, 96.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521979 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887274 s, 118 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79389 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73675 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75941 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74071 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (119s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 10:44:42 (1766418282) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre opts: Stopping clients: oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg228-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg228-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.28,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg228-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.28,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg228-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.28,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/58784/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.28,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg228-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Started clients oleg228-client.virtnet: 192.168.202.128@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Started clients oleg228-client.virtnet: 192.168.202.128@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c1119f7f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c11218ec000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1119f7f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c11218ec000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241946 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023321 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710675 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088481 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102366 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889965 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813007 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103654 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892974 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850714 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095932 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140735 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116333 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231832 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240313 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293091 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282582 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146542 s, 71.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116139 s, 90.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163692 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121332 s, 86.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104802 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220497 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269602 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178818 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244183 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551011 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264837 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217849 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435029 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390572 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296974 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0017879 s, 586 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96895 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9728 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25518 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384112 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341703 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123033 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155313 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14479 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143938 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151992 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159692 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153438 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141419 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141242 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153558 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237554 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238296 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303811 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312857 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108933 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150813 s, 69.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089451 s, 117 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130996 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127479 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567878 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228381 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166783 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498934 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266448 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063576 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267524 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392051 s, 267 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423308 s, 248 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431044 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380477 s, 276 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05166 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72901 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (135s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 10:46:57 (1766418417) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg228-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 10:47:05 (1766418425) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 10:47:08 (1766418428) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg228-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.3778 s, 310 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg228-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.008 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 10:47:58 (1766418478) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124782 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367822 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567771 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467013 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375679 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621647 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580863 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090588 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821613 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860754 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941662 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116474 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12678 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102692 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12488 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113719 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121787 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0083284 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178486 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022119 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105115 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039971 s, 262 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181621 s, 57.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578436 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397678 s, 264 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168226 s, 62.3 MB/sldlm.namespaces.lustre-OST0001-osc-ffff9c11218ec000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00840305 s, 125 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133753 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00649892 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219569 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033825 s, 310 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103965 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302117 s, 347 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95595 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96962 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143715 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873192 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101089 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108935 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110904 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119552 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123182 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107412 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125333 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124948 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113546 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156326 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15105 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193082 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184764 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245353 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25087 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397521 s, 264 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126133 s, 83.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012431 s, 84.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016561 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203204 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104264 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102221 s, 103 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258856 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251996 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255906 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844452 s, 124 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231888 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767008 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958915 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115405 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00534696 s, 196 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg228-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg228-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04609 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97134 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 10:49:04 (1766418544) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (3s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 10:49:07 (1766418547) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (38s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 10:49:45 (1766418585) mds.MDS.mdt.nrs_policies=tbf oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg228-client: oleg228-server: ssh exited with exit code 36 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 oleg228-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg228-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (11s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 10:49:56 (1766418596) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 270386 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 10:50:38 (1766418638) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 10:50:41 (1766418641) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935185 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:50:42.000000000 -0500 Modify: 2025-12-22 10:50:42.000000000 -0500 Change: 2025-12-22 10:50:42.000000000 -0500 Birth: 2025-12-22 10:50:42.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 10:50:54 (1766418654) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 10:50:55 (1766418655) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 10:50:56 (1766418656) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 10:50:57 (1766418657) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 10:50:58 (1766418658) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 10:50:59 (1766418659) total: 2500 open/close in 5.24 seconds: 477.34 ops/second snapshot_time 1766418682.431137800 secs.nsecs start_time 1766418660.944370273 secs.nsecs elapsed_time 21.486767527 secs.nsecs open 2505 samples [usecs] 65 1260 734846 233247440 close 2504 samples [usecs] 67 683 240938 24743780 mknod 2501 samples [usecs] 168 1143 638191 177662531 unlink 2500 samples [usecs] 281 15797 1185441 1368199229 mkdir 2 samples [usecs] 562 861 1423 1057165 rmdir 2 samples [usecs] 1977 3520 5497 16298929 rename 2500 samples [usecs] 256 9917 2901979 4735097659 getattr 3760 samples [usecs] 7 692 74254 2367558 parallel_rename_file 2500 samples [usecs] 256 9917 2901979 4735097659 crossdir_rename 2500 samples [usecs] 256 9917 2901979 4735097659 PASS 81d (24s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 10:51:23 (1766418683) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7542 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7542 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 10:51:26 (1766418686) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 10:51:27 (1766418687) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 10:51:34 (1766418694) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1766418695 Finishing test test1 at 1766418695 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 10:51:37 (1766418697) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 10:51:38 (1766418698) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 10:51:39 (1766418699) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 10:51:40 (1766418700) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5857 0x16e1 0x240000400 1 5820 0x16bc 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5858 0x16e2 0x240000400 1 5821 0x16bd 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 10:51:49 (1766418709) osc.lustre-OST0000-osc-ffff9c1119f7f000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9c11218ec000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c1119f7f000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c11218ec000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0405643 s, 202 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08591 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 288775 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157687 s, 260 kB/s osc.lustre-OST0000-osc-ffff9c1119f7f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c11218ec000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1119f7f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c11218ec000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 10:51:59 (1766418719) llite.lustre-ffff9c1119f7f000.fast_read=0 llite.lustre-ffff9c11218ec000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0159693 s, 131 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9c1119f7f000.fast_read=1 llite.lustre-ffff9c11218ec000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 10:52:14 (1766418734) llite.lustre-ffff9c1119f7f000.fast_read=0 llite.lustre-ffff9c11218ec000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00173034 s, 7.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9c1119f7f000.fast_read=1 llite.lustre-ffff9c11218ec000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 10:52:22 (1766418742) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 10:52:23 (1766418743) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00175753 s, 2.3 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 10:52:26 (1766418746) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0495485 s, 42.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 10:52:28 (1766418748) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.045356 s, 46.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 10:52:30 (1766418750) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00183443 s, 6.5 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 10:52:33 (1766418753) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00171698 s, 2.4 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 10:52:35 (1766418755) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00168789 s, 2.4 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 10:52:37 (1766418757) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00151638 s, 2.7 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 10:52:41 (1766418761) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1766418762 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1766418764 Finishing test test23 at 1766418764 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 10:52:48 (1766418768) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 10:52:49 (1766418769) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 10:53:07 (1766418787) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 10:53:08 (1766418788) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0033663 s, 304 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937720 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:53:08.000000000 -0500 Modify: 2025-12-22 10:53:08.000000000 -0500 Change: 2025-12-22 10:53:08.000000000 -0500 Birth: 2025-12-22 10:53:09.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937720 -rw-r--r-- 1 0 0 1766418789 1766418788 1766418788 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937720 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 10:53:08.000000000 -0500 Modify: 2025-12-22 10:53:08.000000000 -0500 Change: 2025-12-22 10:53:08.000000000 -0500 Birth: 2025-12-22 10:53:09.000000000 -0500 1766418788 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 10:53:10 (1766418790) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 10:53:13 (1766418793) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0963418 s, 109 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5827 0x16c3 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 10:53:17 (1766418797) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.104638 s, 100 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5867 0x16eb 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7542 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 10:53:25 (1766418805) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0026184 s, 3.8 MB/s llite.lustre-ffff9c1119f7f000.hybrid_io=0 llite.lustre-ffff9c11218ec000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.2781 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 10:53:32 (1766418812) 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 192.168.202.128@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 PASS 109 (513s) == sanityn test 110: do not grant another lock on resend ========================================================== 11:02:05 (1766419325) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 11:02:06.000000000 -0500 Modify: 2025-12-22 11:02:06.000000000 -0500 Change: 2025-12-22 11:02:06.000000000 -0500 Birth: 2025-12-22 11:02:06.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg228-client.virtnet oleg228-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 11:03:10 (1766419390) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 11:03:11 (1766419391) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 11:03:12 (1766419392) lustre-OST0000-osc-ffff9c11383ce800 active. lustre-OST0001-osc-ffff9c11383ce800 active. lustre-OST0000-osc-ffff9c11383ce800 active. lustre-OST0001-osc-ffff9c11383ce800 active. MGC192.168.202.128@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 11:03:14 (1766419394) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 11:03:15 (1766419395) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 11:03:16 (1766419396) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 11:03:17 (1766419397) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00509389 s, 4.0 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139676 s, 1.5 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 11:03:21 (1766419401) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0899 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0917 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0961 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.091 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0935 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0971 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1072 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1506 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1565 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1719 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1595 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1664 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.174 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1797 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.246 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2478 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2376 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2396 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1118eff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1118eff800.ost_server_uuid in IDLE state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 6692 sec ============== 11:04:28 (1766419468) === sanityn: start cleanup 11:04:29 (1766419469) === Stopping clients: oleg228-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg228-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 11:05:15 (1766419515) ===