-----============= acceptance-small: sanityn ============----- Fri Jan 9 20:24:00 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg346-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 20:24:16 (1768008256) === Starting client oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Mount client oleg346-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Started clients oleg346-client.virtnet: 192.168.203.146@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg346-client.virtnet: executing check_config_client /mnt/lustre oleg346-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg346-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9aa147d3e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa14a5b9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa147d3e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa14a5b9800.idle_timeout=debug disable quota as required oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 20:24:35 (1768008275) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246729 s, 42.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7536] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 20:24:36 (1768008276) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 20:24:41 (1768008281) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 20:24:44 (1768008284) -rw-r--r-- 1 root root 0 Jan 9 20:24 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 20:24:46 (1768008286) -rw-r--r-- 1 root root 0 Jan 9 20:24 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 20:24:50 (1768008290) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 20:24:53 (1768008293) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (3s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 20:24:56 (1768008296) total 1 -rwxrwxrwx 1 root root 0 Jan 9 20:24 f2a -rwxrwxrwx 1 root root 0 Jan 9 20:24 f2b total 1 -rwxrwxrwx 1 root root 0 Jan 9 20:24 f2a -rwxrwxrwx 1 root root 0 Jan 9 20:24 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00636412 s, 80.5 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 20:25:00 (1768008300) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 20:25:02 (1768008302) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157746 s, 13.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 20:25:05 (1768008305) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 20:25:08 (1768008308) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 20:25:13 (1768008313) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 20:25:16 (1768008316) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 20:25:19 (1768008319) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 20:25:23 (1768008323) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 20:25:27 (1768008327) PASS 9a (4s) == sanityn test 9b: append to striped sparse file ======== 20:25:31 (1768008331) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050531 s, 20.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0465873 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 20:25:35 (1768008335) 1+0 records in 1+0 records out 1 byte copied, 0.00674152 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00928727 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.015705 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00637505 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.006152 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110747 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00848941 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00928 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00471137 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118005 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00728965 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114796 s, 0.1 kB/s PASS 10a (4s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 20:25:39 (1768008339) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00997712 s, 308 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0397321 s, 103 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000362595 s, 8.5 MB/s PASS 10b (4s) == sanityn test 11: execution of file opened for write should return error ============================================================== 20:25:43 (1768008343) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 20:25:46 (1768008346) warning: '-runtime' deprecated, use '-t runtime' instead - link 784 (time 1768008357.90 total 10.00 last 78.36) - link 1635 (time 1768008367.91 total 20.02 last 85.01) - link 2381 (time 1768008377.92 total 30.02 last 74.55) - link 3029 (time 1768008387.93 total 40.03 last 64.74) - link 3787 (time 1768008397.93 total 50.04 last 75.75) - link 4459 (time 1768008407.94 total 60.04 last 67.17) - link 5203 (time 1768008417.94 total 70.04 last 74.39) - link 5988 (time 1768008427.94 total 80.05 last 78.47) - link 6798 (time 1768008437.95 total 90.05 last 80.98) total: 7534 link in 100.01 seconds: 75.34 ops/second using seed 2485240327 running for 100 seconds - stat 10000 (time 1768008357 ; total 9 ; last 9) - stat 20000 (time 1768008366 ; total 18 ; last 9) - stat 30000 (time 1768008375 ; total 27 ; last 9) - stat 40000 (time 1768008385 ; total 37 ; last 10) - stat 50000 (time 1768008394 ; total 46 ; last 9) - stat 57784 (time 1768008405 ; total 57 ; last 11) - stat 60000 (time 1768008409 ; total 61 ; last 4) - stat 65707 (time 1768008420 ; total 72 ; last 11) - stat 70000 (time 1768008427 ; total 79 ; last 7) - stat 75380 (time 1768008438 ; total 90 ; last 11) - stat 80000 (time 1768008446 ; total 98 ; last 8) total: 80798 stats in 100 seconds: 807.979980 stats/second - unlinked 0 (time 1768008398 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile7534) error: No such file or directory total: 7533 unlinks in 102 seconds: 73.852943 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21313) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21315) - No such process PASS 12 (157s) == sanityn test 13: test directory page revocation ======= 20:28:23 (1768008503) f13 f13 PASS 13 (5s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 20:28:28 (1768008508) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 22503 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 20:28:32 (1768008512) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 23082 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 20:28:35 (1768008515) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23660 Terminated $DIR1/$tdir/sleep 60 PASS 14b (4s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 20:28:39 (1768008519) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24242 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 20:28:42 (1768008522) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24824 Terminated $DIR1/$tdir/sleep 60 PASS 14d (5s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 20:28:47 (1768008527) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (34s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 20:29:21 (1768008561) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.212047 s, 39.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.138815 s, 60.4 MB/s Chance of close/open is 1 in 50 Seed set to 8563 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1768008563.666028 write 0x4bc11e thru 0x4c0c2d (0x4b10 bytes) 000002[0] 1768008563.694499 read 0x04197c thru 0x04cadd (0xb162 bytes) 000004[1] 1768008563.727480 read 0x2c276a thru 0x2ca666 (0x7efd bytes) 000006[1] 1768008563.780730 write 0x69a377 thru 0x6a7ec7 (0xdb51 bytes) 000007[0] 1768008563.792485 mapread 0x35e60f thru 0x36c944 (0xe336 bytes) truncating to largest ever: 0x45dcb8 000010[0] 1768008563.974714 trunc from 0x6a7ec8 to 0x45dcb7 (0x24a210 bytes) 000011[0] 1768008564.025852 mapread 0x161a01 thru 0x165bf5 (0x41f5 bytes) 000012[0] 1768008564.040741 mapwrite 0x9e77f5 thru 0x9e9483 (0x1c8f bytes) 000013[0] 1768008564.089260 mapread 0x0dd30d thru 0x0e2a6e (0x5762 bytes) 000014[1] 1768008564.106799 mapwrite 0x94f300 thru 0x95347e (0x417f bytes) 000015[1] 1768008564.155045 write 0x862a6c thru 0x86c58f (0x9b24 bytes) 000016[1] 1768008564.175958 write 0x7b4365 thru 0x7be8d7 (0xa573 bytes) 000018[1] 1768008564.201494 write 0x72e2b3 thru 0x732c60 (0x49ae bytes) 000023[1] 1768008564.232573 read 0x79224c thru 0x79bf6b (0x9d20 bytes) 000025[0] 1768008564.233545 trunc from 0x9e9484 to 0x43b81f (0x5adc64 bytes) 000026[0] 1768008564.291537 read 0x08ba1a thru 0x09687f (0xae66 bytes) 000027[0] 1768008564.321580 read 0x1ed526 thru 0x1f4b5d (0x7638 bytes) 000029[0] 1768008564.346785 read 0x1bf14f thru 0x1cde01 (0xecb3 bytes) 000030[0] 1768008564.376626 read 0x182c18 thru 0x183c34 (0x101d bytes) truncating to largest ever: 0x701cbf 000031[0] 1768008564.397562 trunc from 0x43b820 to 0x701cbe (0x2c649f bytes) truncating to largest ever: 0x9b840c 000032[1] 1768008564.412834 trunc from 0x701cbf to 0x9b840b (0x2b674d bytes) 000033[0] 1768008564.462751 write 0x8eb324 thru 0x8ed204 (0x1ee1 bytes) 000038[0] 1768008564.543936 read 0x4a98b7 thru 0x4b5d9d (0xc4e7 bytes) 000040[0] 1768008564.554126 mapread 0x5bde09 thru 0x5cce66 (0xf05e bytes) 000043[1] 1768008564.598976 mapwrite 0x1dcd87 thru 0x1e244e (0x56c8 bytes) 000045[1] 1768008564.682646 mapwrite 0x4cb6f5 thru 0x4d3f8c (0x8898 bytes) 000047[1] 1768008564.743730 write 0x91d38d thru 0x92378b (0x63ff bytes) 000050[0] 1768008564.783957 write 0x238ce1 thru 0x23ec50 (0x5f70 bytes) 000051[1] 1768008564.822641 read 0x8e06d5 thru 0x8e7f05 (0x7831 bytes) 000052[0] 1768008564.842415 mapread 0x5c50bd thru 0x5d062e (0xb572 bytes) 000060[1] 1768008564.892290 write 0x67c88c thru 0x6887b5 (0xbf2a bytes) 000065[1] 1768008564.948605 trunc from 0x9b840c to 0x8f6afc (0xc190f bytes) 000066[1] 1768008564.989441 trunc from 0x8f6afd to 0x96d54b (0x76a4f bytes) 000067[1] 1768008565.016713 read 0x8651d9 thru 0x86c253 (0x707b bytes) 000069[1] 1768008565.034855 write 0x672885 thru 0x67f276 (0xc9f2 bytes) 000072[0] 1768008565.050771 mapwrite 0x662763 thru 0x66469c (0x1f3a bytes) 000073[0] 1768008565.106591 mapwrite 0x67bc6d thru 0x6868f8 (0xac8c bytes) 000076[0] 1768008565.193161 mapwrite 0x73260a thru 0x736897 (0x428e bytes) 000077[0] 1768008565.255730 mapwrite 0x9eec8e thru 0x9f3ddf (0x5152 bytes) 000079[1] 1768008565.337489 write 0x7e159f thru 0x7e2e75 (0x18d7 bytes) 000081[0] 1768008565.382557 write 0x988494 thru 0x99678a (0xe2f7 bytes) 000086[1] 1768008565.455409 write 0x6a4330 thru 0x6b3954 (0xf625 bytes) 000087[1] 1768008565.470649 trunc from 0x9f3de0 to 0x350256 (0x6a3b89 bytes) 000088[1] 1768008565.508366 trunc from 0x350257 to 0x8a011f (0x54fec9 bytes) 000089[1] 1768008565.533940 mapread 0x8901a6 thru 0x8934cd (0x3328 bytes) 000091[1] 1768008565.548286 mapread 0x3f0348 thru 0x3f4fda (0x4c93 bytes) 000092[0] 1768008565.583356 write 0x5a58f6 thru 0x5ad76e (0x7e79 bytes) 000096[1] 1768008565.656603 write 0x046208 thru 0x04b292 (0x508b bytes) 000097[0] 1768008565.677041 write 0x3fd4cf thru 0x40cb4b (0xf67d bytes) 000098[0] 1768008565.712901 trunc from 0x8a0120 to 0x2a9b46 (0x5f65d9 bytes) 000100[1] 1768008565.746645 write 0x9e2ff0 thru 0x9e6258 (0x3269 bytes) 000102[1] 1768008565.773094 read 0x0215b9 thru 0x0314c8 (0xff10 bytes) 000106[1] 1768008565.830851 write 0x748baf thru 0x755643 (0xca95 bytes) 000107[0] 1768008565.845509 write 0x7d4deb thru 0x7d7552 (0x2768 bytes) 000108[1] 1768008565.865936 mapread 0x0f3c02 thru 0x0f93f0 (0x57ef bytes) 000109[1] 1768008565.900977 mapread 0x29a4c9 thru 0x2a28c5 (0x83fd bytes) 000110[0] 1768008565.923683 mapread 0x1ccc22 thru 0x1d4a56 (0x7e35 bytes) 000111[1] 1768008565.960418 trunc from 0x9e6259 to 0x0e5555 (0x900d03 bytes) 000115[0] 1768008566.044638 read 0x007b26 thru 0x00f1eb (0x76c6 bytes) 000117[1] 1768008566.077619 read 0x0a4e9c thru 0x0ae3c5 (0x952a bytes) 000118[0] 1768008566.097309 mapread 0x0d4448 thru 0x0d5dea (0x19a3 bytes) 000119[1] 1768008566.101592 write 0x174411 thru 0x17b945 (0x7535 bytes) 000121[0] 1768008566.120934 write 0x3462b8 thru 0x35374a (0xd493 bytes) 000122[1] 1768008566.136405 mapwrite 0x3c5b4a thru 0x3cfa94 (0x9f4b bytes) 000123[0] 1768008566.228938 mapwrite 0x9ed6fd thru 0x9f446f (0x6d73 bytes) 000124[1] 1768008566.293020 trunc from 0x9f4470 to 0x99f151 (0x5531e bytes) 000125[0] 1768008566.321380 mapread 0x263f7a thru 0x26f1b9 (0xb240 bytes) 000128[1] 1768008566.351111 mapread 0x40ee80 thru 0x417573 (0x86f4 bytes) 000129[0] 1768008566.399339 mapread 0x559654 thru 0x563a81 (0xa42e bytes) 000130[0] 1768008566.436774 read 0x341fc8 thru 0x343f15 (0x1f4e bytes) 000132[1] 1768008566.445127 trunc from 0x99f152 to 0x18c47a (0x812cd7 bytes) 000135[0] 1768008566.515969 trunc from 0x18c47b to 0x401026 (0x274bac bytes) 000136[0] 1768008566.564622 write 0x62b68a thru 0x62fee3 (0x485a bytes) 000137[1] 1768008566.568876 read 0x5573e5 thru 0x561849 (0xa465 bytes) 000138[0] 1768008566.596671 trunc from 0x62fee4 to 0x93cf8b (0x30d0a8 bytes) 000140[0] 1768008566.621528 mapwrite 0x9999fe thru 0x99b1f5 (0x17f8 bytes) 000143[0] 1768008566.711534 mapwrite 0x429a90 thru 0x42acaa (0x121b bytes) 000144[0] 1768008566.772137 mapread 0x3ea4f2 thru 0x3f628f (0xbd9e bytes) 000145[0] 1768008566.799709 write 0x07f569 thru 0x084656 (0x50ee bytes) 000149[0] 1768008566.842703 mapwrite 0x60c9c5 thru 0x618532 (0xbb6e bytes) 000150[0] 1768008566.888113 read 0x72e644 thru 0x7374ad (0x8e6a bytes) 000151[1] 1768008566.903733 mapread 0x7b9759 thru 0x7c4b41 (0xb3e9 bytes) 000155[1] 1768008566.982828 mapread 0x4c0628 thru 0x4c53d0 (0x4da9 bytes) 000156[0] 1768008567.025941 read 0x2a54e8 thru 0x2abe7d (0x6996 bytes) 000158[0] 1768008567.055612 trunc from 0x99b1f6 to 0x4b820c (0x4e2fe9 bytes) 000160[0] 1768008567.085937 write 0x0d8804 thru 0x0e09dc (0x81d9 bytes) 000161[1] 1768008567.108988 trunc from 0x4b820d to 0x9a5d58 (0x4edb4c bytes) 000162[0] 1768008567.179334 trunc from 0x9a5d59 to 0x8330a1 (0x172cb7 bytes) 000165[0] 1768008567.246978 mapread 0x0b225a thru 0x0b8986 (0x672d bytes) 000167[0] 1768008567.292666 mapwrite 0x1c6c60 thru 0x1d105c (0xa3fd bytes) 000169[0] 1768008567.368961 trunc from 0x8330a2 to 0x117a33 (0x71b66e bytes) 000170[1] 1768008567.391445 write 0x6b3e94 thru 0x6bff6e (0xc0db bytes) 000171[1] 1768008567.415129 trunc from 0x6bff6f to 0x488e79 (0x2370f5 bytes) 000173[0] 1768008567.460136 mapread 0x473ac4 thru 0x4792ee (0x582b bytes) 000178[1] 1768008567.549824 trunc from 0x488e7a to 0x65a420 (0x1d15a7 bytes) 000181[0] 1768008567.609770 write 0x3a6e75 thru 0x3a8e8f (0x201b bytes) 000185[1] 1768008567.656435 trunc from 0x65a421 to 0x4b217c (0x1a82a4 bytes) 000188[0] 1768008567.690961 mapwrite 0x009457 thru 0x014aa2 (0xb64c bytes) 000190[1] 1768008567.766770 mapread 0x02cd17 thru 0x036f6f (0xa259 bytes) 000191[1] 1768008567.793769 mapread 0x016b55 thru 0x024068 (0xd514 bytes) 000192[1] 1768008567.813942 mapwrite 0x5ea668 thru 0x5ede8d (0x3826 bytes) 000193[0] 1768008567.897198 write 0x9299fe thru 0x931f54 (0x8557 bytes) 000197[0] 1768008567.987080 write 0x547e37 thru 0x54dd86 (0x5f50 bytes) 000198[0] 1768008568.025343 trunc from 0x931f55 to 0x710792 (0x2217c2 bytes) 000200[0] 1768008568.080171 trunc from 0x710793 to 0x76e5bd (0x5de2b bytes) 000201[1] 1768008568.098774 write 0x6edcd8 thru 0x6f1038 (0x3361 bytes) 000204[0] 1768008568.164400 mapwrite 0x2d197f thru 0x2d7443 (0x5ac5 bytes) 000205[0] 1768008568.214514 mapread 0x250340 thru 0x2509d1 (0x692 bytes) 000212[1] 1768008568.322169 read 0x4d496a thru 0x4db45c (0x6af3 bytes) 000215[1] 1768008568.357016 trunc from 0x76e5be to 0x7f4669 (0x860ac bytes) 000216[1] 1768008568.386989 mapread 0x167043 thru 0x16f538 (0x84f6 bytes) 000218[0] 1768008568.434744 read 0x1780fc thru 0x179134 (0x1039 bytes) 000219[0] 1768008568.458010 trunc from 0x7f466a to 0x4a4281 (0x3503e8 bytes) 000223[0] 1768008568.538899 trunc from 0x4a4282 to 0x86866e (0x3c43ed bytes) 000226[0] 1768008568.558380 mapwrite 0x2a8aa8 thru 0x2b202d (0x9586 bytes) 000229[0] 1768008568.660933 trunc from 0x86866f to 0x0eecfb (0x779973 bytes) 000234[0] 1768008568.728191 write 0x832033 thru 0x83b017 (0x8fe5 bytes) 000238[0] 1768008568.773580 trunc from 0x83b018 to 0x5ac211 (0x28ee06 bytes) 000240[0] 1768008568.825877 write 0x7d17d0 thru 0x7db021 (0x9852 bytes) 000241[0] 1768008568.830558 read 0x35b371 thru 0x3698bb (0xe54b bytes) 000242[0] 1768008568.852546 mapwrite 0x989b53 thru 0x98a517 (0x9c5 bytes) 000244[0] 1768008568.952995 mapwrite 0x252fda thru 0x258d6c (0x5d93 bytes) 000245[0] 1768008569.038787 mapread 0x100e5a thru 0x101549 (0x6f0 bytes) 000247[1] 1768008569.084264 read 0x273d97 thru 0x27c91e (0x8b88 bytes) 000248[0] 1768008569.098081 mapread 0x556e63 thru 0x55f55e (0x86fc bytes) 000249[0] 1768008569.112726 trunc from 0x98a518 to 0x667e72 (0x3226a5 bytes) 000250[1] 1768008569.142566 mapwrite 0x4d0657 thru 0x4dc047 (0xb9f1 bytes) 000254[1] 1768008569.278960 mapwrite 0x57c0c7 thru 0x57de90 (0x1dca bytes) 000255[1] 1768008569.348190 write 0x13a739 thru 0x144862 (0xa12a bytes) 000258[0] 1768008569.377678 mapread 0x09004c thru 0x09f4ec (0xf4a1 bytes) 000259[0] 1768008569.450037 mapread 0x34ac13 thru 0x358f00 (0xe2ee bytes) 000261[0] 1768008569.518503 mapread 0x018a6a thru 0x01f895 (0x6e2c bytes) 000262[0] 1768008569.545272 mapread 0x0a7072 thru 0x0affba (0x8f49 bytes) 000264[1] 1768008569.567124 mapread 0x572c8b thru 0x58275f (0xfad5 bytes) 000267[1] 1768008569.639320 read 0x42a5cf thru 0x42bfa4 (0x19d6 bytes) 000274[1] 1768008569.715128 trunc from 0x667e73 to 0x37316e (0x2f4d04 bytes) 000275[0] 1768008569.749836 mapwrite 0x903963 thru 0x90e671 (0xad0f bytes) 000276[0] 1768008569.877488 write 0x612cdb thru 0x621ce2 (0xf008 bytes) 000285[0] 1768008569.959512 read 0x8c13c0 thru 0x8c2de8 (0x1a29 bytes) 000286[1] 1768008569.960198 mapwrite 0x9e9b81 thru 0x9f52d6 (0xb756 bytes) 000288[0] 1768008570.098099 mapwrite 0x479a25 thru 0x4825d5 (0x8bb1 bytes) 000290[0] 1768008570.184454 trunc from 0x9f52d7 to 0x17b7c0 (0x879b16 bytes) 000292[0] 1768008570.231150 mapread 0x05d27e thru 0x05ff32 (0x2cb5 bytes) 000294[1] 1768008570.245334 read 0x06d4ae thru 0x06dcb2 (0x805 bytes) 000300[0] 1768008570.314386 read 0x1041f1 thru 0x111096 (0xcea6 bytes) 000301[1] 1768008570.347545 write 0x6189ce thru 0x61ee1b (0x644e bytes) 000303[0] 1768008570.375577 read 0x2e8053 thru 0x2ee6fa (0x66a8 bytes) 000305[1] 1768008570.406770 trunc from 0x61ee1c to 0x7000c6 (0xe12ab bytes) 000306[1] 1768008570.452618 mapwrite 0x52be70 thru 0x53ace2 (0xee73 bytes) 000307[1] 1768008570.529963 trunc from 0x7000c7 to 0x4b4189 (0x24bf3d bytes) 000308[0] 1768008570.546747 mapread 0x142aa7 thru 0x1436b3 (0xc0d bytes) 000309[1] 1768008570.577383 read 0x44c94f thru 0x44cc22 (0x2d4 bytes) 000316[0] 1768008570.600174 mapread 0x3b59cd thru 0x3b5b13 (0x147 bytes) 000317[1] 1768008570.611706 write 0x80da31 thru 0x80e546 (0xb16 bytes) 000318[0] 1768008570.633858 mapread 0x36eee3 thru 0x37da9e (0xebbc bytes) 000322[0] 1768008570.701123 mapread 0x64d6af thru 0x656f88 (0x98da bytes) 000324[0] 1768008570.772131 trunc from 0x80e547 to 0x25146d (0x5bd0d9 bytes) 000325[0] 1768008570.823781 trunc from 0x25146e to 0x3e3c62 (0x1927f5 bytes) 000326[1] 1768008570.834873 write 0x73fc0a thru 0x74f524 (0xf91b bytes) 000327[1] 1768008570.855804 write 0x965034 thru 0x974a8a (0xfa57 bytes) 000329[0] 1768008570.862500 mapwrite 0x0ca659 thru 0x0d729f (0xcc47 bytes) 000332[1] 1768008570.988036 mapwrite 0x5e2552 thru 0x5ebd39 (0x97e8 bytes) 000333[0] 1768008571.062454 read 0x6aafa4 thru 0x6b4745 (0x97a2 bytes) 000334[1] 1768008571.078521 mapwrite 0x63eb25 thru 0x647639 (0x8b15 bytes) 000336[0] 1768008571.168235 mapwrite 0x427b6e thru 0x42dd82 (0x6215 bytes) 000337[1] 1768008571.234047 write 0x0ac75e thru 0x0b51f1 (0x8a94 bytes) 000341[1] 1768008571.292110 mapwrite 0x7b7d6e thru 0x7bf4b6 (0x7749 bytes) 000342[0] 1768008571.424193 write 0x2c15e2 thru 0x2c307d (0x1a9c bytes) 000345[0] 1768008571.488643 read 0x877ec8 thru 0x87a089 (0x21c2 bytes) 000352[0] 1768008571.550185 mapwrite 0x00f3c8 thru 0x01a26b (0xaea4 bytes) 000353[0] 1768008571.628691 write 0x294bc1 thru 0x29ff89 (0xb3c9 bytes) 000354[1] 1768008571.648301 mapread 0x71bcc5 thru 0x7289a7 (0xcce3 bytes) 000355[0] 1768008571.687021 write 0x27c456 thru 0x27d556 (0x1101 bytes) 000357[1] 1768008571.714096 mapread 0x3d5b2b thru 0x3ddc76 (0x814c bytes) 000360[0] 1768008571.754849 trunc from 0x974a8b to 0x258fcd (0x71babd bytes) 000361[1] 1768008571.785419 trunc from 0x258fce to 0x859986 (0x6009b9 bytes) 000362[0] 1768008571.824879 mapread 0x031e19 thru 0x03ff65 (0xe14d bytes) 000363[1] 1768008571.856245 mapread 0x23adcb thru 0x23df56 (0x318c bytes) 000364[0] 1768008571.873180 read 0x67fca6 thru 0x6823a5 (0x2700 bytes) 000365[0] 1768008571.888818 write 0x4aacc8 thru 0x4b3646 (0x897f bytes) 000371[0] 1768008571.925250 write 0x4c0b90 thru 0x4cddc8 (0xd239 bytes) 000372[1] 1768008571.947383 write 0x6b6da5 thru 0x6c49f9 (0xdc55 bytes) 000373[0] 1768008571.975595 read 0x432c54 thru 0x43a4cf (0x787c bytes) 000374[1] 1768008571.989061 mapread 0x3c8f32 thru 0x3d89bd (0xfa8c bytes) 000375[1] 1768008572.005989 trunc from 0x859987 to 0x0c6ea2 (0x792ae4 bytes) 000378[0] 1768008572.044481 trunc from 0x0c6ea3 to 0x7a3aef (0x6dcc4d bytes) 000382[1] 1768008572.076206 write 0x39da15 thru 0x3aa023 (0xc60f bytes) 000386[0] 1768008572.125514 read 0x4e8270 thru 0x4ef782 (0x7513 bytes) 000387[1] 1768008572.136248 mapwrite 0x20352d thru 0x209109 (0x5bdd bytes) 000388[1] 1768008572.182015 trunc from 0x7a3af0 to 0x1dc742 (0x5c73ad bytes) 000391[0] 1768008572.215911 mapwrite 0x0af97a thru 0x0bde3a (0xe4c1 bytes) 000395[0] 1768008572.309478 mapread 0x106d63 thru 0x109723 (0x29c1 bytes) 000397[0] 1768008572.346955 mapwrite 0x047fe1 thru 0x04cdfc (0x4e1c bytes) 000398[0] 1768008572.395186 mapwrite 0x1b479d thru 0x1c0daa (0xc60e bytes) 000399[0] 1768008572.444647 read 0x181572 thru 0x18f047 (0xdad6 bytes) 000402[0] 1768008572.478052 mapread 0x0e0632 thru 0x0ede0d (0xd7dc bytes) 000405[1] 1768008572.518318 write 0x2ac03d thru 0x2b113e (0x5102 bytes) 000409[1] 1768008572.542086 trunc from 0x2b113f to 0x0ca11c (0x1e7022 bytes) 000410[1] 1768008572.564250 mapwrite 0x79bafa thru 0x7a3df8 (0x82ff bytes) 000413[1] 1768008572.630112 mapwrite 0x39ec57 thru 0x3a90d9 (0xa483 bytes) 000414[1] 1768008572.710858 trunc from 0x7a3df9 to 0x805886 (0x61a8e bytes) 000415[0] 1768008572.736397 trunc from 0x805887 to 0x4a503a (0x36084c bytes) 000416[0] 1768008572.784254 write 0x292832 thru 0x29382a (0xff9 bytes) 000419[1] 1768008572.843448 read 0x099a1a thru 0x09a550 (0xb37 bytes) 000420[1] 1768008572.873084 read 0x014627 thru 0x022270 (0xdc4a bytes) 000421[0] 1768008572.916547 read 0x39cc5c thru 0x39e824 (0x1bc9 bytes) 000423[1] 1768008572.927033 write 0x267234 thru 0x2730c2 (0xbe8f bytes) 000424[0] 1768008572.974460 write 0x20e68c thru 0x213c0d (0x5582 bytes) 000428[1] 1768008573.054536 read 0x000125 thru 0x007d2b (0x7c07 bytes) 000429[1] 1768008573.082324 write 0x4b4f3c thru 0x4be2e3 (0x93a8 bytes) 000430[1] 1768008573.097074 write 0x0ed3dc thru 0x0fa34b (0xcf70 bytes) 000435[0] 1768008573.142294 trunc from 0x4be2e4 to 0x84e37f (0x39009c bytes) 000436[1] 1768008573.185742 read 0x0fe2f7 thru 0x1004aa (0x21b4 bytes) 000437[0] 1768008573.212928 mapwrite 0x3bef2c thru 0x3cc288 (0xd35d bytes) 000441[1] 1768008573.294006 mapwrite 0x7bf267 thru 0x7c36a3 (0x443d bytes) 000446[0] 1768008573.396788 trunc from 0x84e380 to 0x957c29 (0x1098aa bytes) 000448[1] 1768008573.437401 trunc from 0x957c2a to 0x407656 (0x5505d3 bytes) 000449[0] 1768008573.485410 mapread 0x33f90a thru 0x34399e (0x4095 bytes) 000451[0] 1768008573.523007 write 0x7b2c03 thru 0x7c25e7 (0xf9e5 bytes) 000454[1] 1768008573.585754 trunc from 0x7c25e8 to 0x737df0 (0x8a7f7 bytes) 000455[0] 1768008573.629675 mapread 0x4a2632 thru 0x4a55e6 (0x2fb5 bytes) 000456[0] 1768008573.667919 mapwrite 0x6c84e4 thru 0x6d7cdf (0xf7fc bytes) 000457[1] 1768008573.746578 read 0x649a3d thru 0x64ebb4 (0x5178 bytes) 000461[0] 1768008573.775363 write 0x05c82f thru 0x066e5a (0xa62c bytes) 000462[1] 1768008573.801745 mapread 0x39507a thru 0x396485 (0x140c bytes) 000463[1] 1768008573.831015 trunc from 0x737df1 to 0x433e5d (0x303f93 bytes) 000465[0] 1768008573.906496 trunc from 0x433e5e to 0x89737e (0x463521 bytes) 000467[0] 1768008573.954020 mapread 0x4749df thru 0x478b1e (0x4140 bytes) 000468[1] 1768008573.990701 trunc from 0x89737f to 0x6d62ef (0x1c108f bytes) 000469[0] 1768008574.029290 mapread 0x11e22c thru 0x120d4b (0x2b20 bytes) 000470[0] 1768008574.071509 trunc from 0x6d62f0 to 0x3abba8 (0x32a747 bytes) 000472[1] 1768008574.099703 write 0x6f2b6e thru 0x6f7653 (0x4ae6 bytes) 000474[0] 1768008574.132427 write 0x3fa0ee thru 0x4074d0 (0xd3e3 bytes) 000477[0] 1768008574.176711 mapread 0x07c9f6 thru 0x08b622 (0xec2d bytes) 000480[0] 1768008574.227851 mapread 0x5b1900 thru 0x5c1695 (0xfd96 bytes) 000481[0] 1768008574.269165 trunc from 0x6f7654 to 0x94b42b (0x253dd8 bytes) 000483[0] 1768008574.295955 read 0x40d5f8 thru 0x410438 (0x2e41 bytes) truncating to largest ever: 0x9c0f89 000488[1] 1768008574.381186 trunc from 0x94b42c to 0x9c0f88 (0x75b5d bytes) 000489[1] 1768008574.435405 read 0x6649b4 thru 0x66affe (0x664b bytes) 000490[1] 1768008574.445404 write 0x3bfdca thru 0x3cb203 (0xb43a bytes) 000491[0] 1768008574.453431 mapread 0x7bace6 thru 0x7c002f (0x534a bytes) 000492[0] 1768008574.485744 read 0x9134fd thru 0x9210d7 (0xdbdb bytes) 000495[0] 1768008574.513373 trunc from 0x9c0f89 to 0x1aa3dc (0x816bac bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8574 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005[0] 1768008575.043558 mapwrite 0x20e000 thru 0x21bd48 (0xdd49 bytes) 000008[0] 1768008575.108837 mapwrite 0x9fb000 thru 0x9ff130 (0x4131 bytes) 000009[1] 1768008575.173659 read_OD 0x499000 thru 0x49bfff (0x3000 bytes) 000010[1] 1768008575.185315 read_OD 0x302000 thru 0x307fff (0x6000 bytes) 000012[0] 1768008575.195926 mapwrite 0x602000 thru 0x6022f8 (0x2f9 bytes) 000015[1] 1768008575.227855 write_OD 0x6c7000 thru 0x6d5fff (0xf000 bytes) 000017[0] 1768008575.254455 mapwrite 0x23a000 thru 0x23ec2f (0x4c30 bytes) 000019[1] 1768008575.330717 read_OD 0x80f000 thru 0x819fff (0xb000 bytes) 000021[0] 1768008575.350565 write 0x02f000 thru 0x036094 (0x7095 bytes) 000022[0] 1768008575.379024 mapread 0x8b8000 thru 0x8c6b70 (0xeb71 bytes) 000024[0] 1768008575.406268 read 0x3c8000 thru 0x3ce850 (0x6851 bytes) 000025[1] 1768008575.411107 read_OD 0x6df000 thru 0x6e6fff (0x8000 bytes) 000026[1] 1768008575.432472 mapwrite 0x951000 thru 0x9585c6 (0x75c7 bytes) 000028[0] 1768008575.514739 read 0x0d8000 thru 0x0e77e1 (0xf7e2 bytes) 000037[1] 1768008575.536368 mapwrite 0x134000 thru 0x13f059 (0xb05a bytes) 000038[0] 1768008575.596303 read 0x00a000 thru 0x018986 (0xe987 bytes) 000040[1] 1768008575.608178 mapwrite 0x037000 thru 0x0421d4 (0xb1d5 bytes) 000041[1] 1768008575.680164 mapread 0x8d5000 thru 0x8d57e2 (0x7e3 bytes) 000043[0] 1768008575.696688 read 0x4b9000 thru 0x4c2d67 (0x9d68 bytes) truncating to largest ever: 0x820da7 000045[0] 1768008575.714831 trunc from 0x9ff131 to 0x820da6 (0x1de38a bytes) 000046[0] 1768008575.739157 write 0x985000 thru 0x9944a0 (0xf4a1 bytes) 000048[0] 1768008575.755310 mapread 0x0de000 thru 0x0e0931 (0x2932 bytes) 000049[0] 1768008575.765836 read 0x0b0000 thru 0x0be97d (0xe97e bytes) 000051[0] 1768008575.784501 read 0x5f6000 thru 0x5f9b4a (0x3b4b bytes) 000052[0] 1768008575.793698 mapwrite 0x801000 thru 0x80bfe1 (0xafe2 bytes) 000053[0] 1768008575.836753 read 0x940000 thru 0x94d40e (0xd40f bytes) 000054[0] 1768008575.870693 read 0x8e5000 thru 0x8f02f1 (0xb2f2 bytes) 000055[0] 1768008575.891787 trunc from 0x9944a1 to 0x0ad03e (0x8e7462 bytes) 000058[0] 1768008575.927797 trunc from 0x0ad03f to 0x4286e1 (0x37b6a3 bytes) 000059[1] 1768008575.945458 write_OD 0x8cd000 thru 0x8d5fff (0x9000 bytes) 000063[1] 1768008575.972678 trunc from 0x8d6000 to 0x3fd342 (0x4d8cbd bytes) 000065[0] 1768008576.000290 trunc from 0x3fd343 to 0x6b71be (0x2b9e7c bytes) 000066[1] 1768008576.022193 trunc from 0x6b71bf to 0x358a73 (0x35e74b bytes) 000067[0] 1768008576.042687 mapread 0x0ee000 thru 0x0f4448 (0x6449 bytes) 000070[0] 1768008576.071579 write 0x084000 thru 0x090cb4 (0xccb5 bytes) 000072[0] 1768008576.100124 write 0x0cd000 thru 0x0d0015 (0x3016 bytes) 000074[1] 1768008576.117686 read_OD 0x2b7000 thru 0x2c4fff (0xe000 bytes) 000075[1] 1768008576.131588 read_OD 0x315000 thru 0x31afff (0x6000 bytes) truncating to largest ever: 0x9b7bda 000077[1] 1768008576.138920 trunc from 0x358a74 to 0x9b7bd9 (0x65f166 bytes) 000078[1] 1768008576.173490 trunc from 0x9b7bda to 0x808ac0 (0x1af119 bytes) 000079[0] 1768008576.185861 mapread 0x06f000 thru 0x07a271 (0xb272 bytes) 000081[1] 1768008576.207334 trunc from 0x808ac1 to 0x914488 (0x10b9c8 bytes) 000082[1] 1768008576.224428 mapwrite 0x128000 thru 0x13392d (0xb92e bytes) 000083[1] 1768008576.293087 read_OD 0x6a6000 thru 0x6b1fff (0xc000 bytes) 000084[0] 1768008576.314481 mapwrite 0x19f000 thru 0x1a05ec (0x15ed bytes) 000086[0] 1768008576.365623 mapwrite 0x8c8000 thru 0x8d5c3d (0xdc3e bytes) 000088[1] 1768008576.436011 mapread 0x1bb000 thru 0x1ca8dd (0xf8de bytes) 000089[1] 1768008576.472858 trunc from 0x914489 to 0x7a07da (0x173cae bytes) 000093[1] 1768008576.506179 write_OD 0x8ff000 thru 0x903fff (0x5000 bytes) 000094[0] 1768008576.530680 read 0x0fb000 thru 0x0fbab7 (0xab8 bytes) 000096[0] 1768008576.538049 mapwrite 0x122000 thru 0x12ef53 (0xcf54 bytes) 000100[0] 1768008576.623837 mapread 0x09a000 thru 0x09fc68 (0x5c69 bytes) 000101[0] 1768008576.631653 write 0x21d000 thru 0x22b27b (0xe27c bytes) 000103[1] 1768008576.648470 trunc from 0x904000 to 0x6cb921 (0x2386de bytes) 000104[0] 1768008576.699376 mapread 0x1c3000 thru 0x1d0328 (0xd329 bytes) 000106[1] 1768008576.749588 mapread 0x1df000 thru 0x1e754a (0x854b bytes) 000108[0] 1768008576.769975 trunc from 0x6cb922 to 0x9258b5 (0x259f94 bytes) 000112[0] 1768008576.857770 write 0x824000 thru 0x8333d0 (0xf3d1 bytes) 000113[1] 1768008576.874407 mapwrite 0x7c6000 thru 0x7c6b8b (0xb8c bytes) 000114[1] 1768008576.946381 trunc from 0x9258b6 to 0x3b0fb3 (0x574902 bytes) 000119[1] 1768008576.979786 mapread 0x2d1000 thru 0x2dd8e4 (0xc8e5 bytes) 000122[0] 1768008577.036520 mapread 0x1c0000 thru 0x1c4ecd (0x4ece bytes) 000123[0] 1768008577.076267 read 0x311000 thru 0x31f286 (0xe287 bytes) 000125[0] 1768008577.133014 trunc from 0x3b0fb4 to 0x9971f4 (0x5e6241 bytes) 000128[1] 1768008577.164414 write_OD 0x205000 thru 0x209fff (0x5000 bytes) 000130[0] 1768008577.190072 mapread 0x30e000 thru 0x317986 (0x9987 bytes) 000131[1] 1768008577.228907 mapread 0x27d000 thru 0x288b34 (0xbb35 bytes) 000133[1] 1768008577.240395 write_OD 0x2e5000 thru 0x2e9fff (0x5000 bytes) 000137[0] 1768008577.267811 mapread 0x38c000 thru 0x396e85 (0xae86 bytes) 000144[0] 1768008577.327936 mapwrite 0x984000 thru 0x984123 (0x124 bytes) 000146[1] 1768008577.442468 mapread 0x724000 thru 0x728f6c (0x4f6d bytes) 000149[0] 1768008577.493307 trunc from 0x9971f5 to 0x633f93 (0x363261 bytes) 000150[1] 1768008577.544700 trunc from 0x633f94 to 0x10e90d (0x525686 bytes) 000157[1] 1768008577.595760 trunc from 0x10e90e to 0x4128b6 (0x303fa9 bytes) 000158[0] 1768008577.612127 mapread 0x31a000 thru 0x320d31 (0x6d32 bytes) 000159[1] 1768008577.654385 mapread 0x338000 thru 0x34664e (0xe64f bytes) 000160[1] 1768008577.671035 mapread 0x29b000 thru 0x2a90db (0xe0dc bytes) 000161[1] 1768008577.682643 write_OD 0x05f000 thru 0x067fff (0x9000 bytes) 000164[1] 1768008577.710020 mapwrite 0x88b000 thru 0x88fb3e (0x4b3f bytes) 000165[0] 1768008577.776808 read 0x2b5000 thru 0x2b90ad (0x40ae bytes) 000171[1] 1768008577.799732 read_OD 0x319000 thru 0x322fff (0xa000 bytes) 000173[0] 1768008577.806479 write 0x687000 thru 0x687c5e (0xc5f bytes) 000174[0] 1768008577.829035 write 0x835000 thru 0x8405cf (0xb5d0 bytes) 000176[1] 1768008577.860470 mapwrite 0x091000 thru 0x097dbc (0x6dbd bytes) 000179[1] 1768008577.955129 read_OD 0x589000 thru 0x58efff (0x6000 bytes) 000181[1] 1768008577.969831 read_OD 0x11a000 thru 0x123fff (0xa000 bytes) 000182[0] 1768008577.982552 read 0x172000 thru 0x17d709 (0xb70a bytes) 000185[0] 1768008577.991384 write 0x2b7000 thru 0x2c6b0b (0xfb0c bytes) 000187[1] 1768008578.030702 write_OD 0x55c000 thru 0x569fff (0xe000 bytes) 000188[1] 1768008578.061848 trunc from 0x88fb3f to 0x5c6649 (0x2c94f5 bytes) 000189[1] 1768008578.101638 read_OD 0x196000 thru 0x19afff (0x5000 bytes) 000190[0] 1768008578.111790 read 0x155000 thru 0x15b075 (0x6076 bytes) 000192[0] 1768008578.125852 mapwrite 0x309000 thru 0x30fd8d (0x6d8e bytes) 000194[1] 1768008578.176029 read_OD 0x2d6000 thru 0x2d8fff (0x3000 bytes) 000195[1] 1768008578.192035 mapwrite 0x55f000 thru 0x56cf3a (0xdf3b bytes) 000197[1] 1768008578.292431 read_OD 0x159000 thru 0x159fff (0x1000 bytes) 000198[1] 1768008578.306484 trunc from 0x5c664a to 0x345d8e (0x2808bb bytes) 000202[1] 1768008578.357391 mapwrite 0x39c000 thru 0x3a15fe (0x55ff bytes) 000203[1] 1768008578.458609 mapread 0x2dc000 thru 0x2dc57b (0x57c bytes) 000205[0] 1768008578.471855 mapread 0x0e9000 thru 0x0f88a5 (0xf8a6 bytes) 000206[0] 1768008578.515392 write 0x2a4000 thru 0x2a4678 (0x679 bytes) 000207[1] 1768008578.523990 mapwrite 0x0c0000 thru 0x0ccaeb (0xcaec bytes) 000211[0] 1768008578.642011 trunc from 0x3a15ff to 0x6df3ab (0x33ddad bytes) 000213[0] 1768008578.681997 mapread 0x193000 thru 0x198b1c (0x5b1d bytes) 000215[0] 1768008578.724329 mapwrite 0x426000 thru 0x433170 (0xd171 bytes) 000216[1] 1768008578.807369 mapread 0x41b000 thru 0x425cca (0xaccb bytes) 000218[1] 1768008578.840880 write_OD 0x31f000 thru 0x327fff (0x9000 bytes) truncating to largest ever: 0x9c1c5f 000221[1] 1768008578.880509 trunc from 0x6df3ac to 0x9c1c5e (0x2e28b3 bytes) 000224[0] 1768008578.949874 mapwrite 0x5ac000 thru 0x5b7096 (0xb097 bytes) 000228[0] 1768008579.036708 mapwrite 0x3f8000 thru 0x3ff325 (0x7326 bytes) 000230[1] 1768008579.086954 write_OD 0x094000 thru 0x0a0fff (0xd000 bytes) 000231[1] 1768008579.114489 mapread 0x32a000 thru 0x330ce1 (0x6ce2 bytes) 000232[0] 1768008579.127310 mapwrite 0x682000 thru 0x68dbac (0xbbad bytes) 000233[1] 1768008579.178574 trunc from 0x9c1c5f to 0x78053f (0x24171f bytes) 000235[0] 1768008579.208451 read 0x070000 thru 0x074d81 (0x4d82 bytes) 000240[1] 1768008579.220425 write_OD 0x2d3000 thru 0x2d5fff (0x3000 bytes) 000241[0] 1768008579.263768 mapwrite 0x2e3000 thru 0x2ed9e1 (0xa9e2 bytes) 000243[1] 1768008579.318696 read_OD 0x208000 thru 0x208fff (0x1000 bytes) 000244[1] 1768008579.329895 trunc from 0x780540 to 0x4661da (0x31a365 bytes) 000245[0] 1768008579.358505 mapwrite 0x188000 thru 0x18def9 (0x5efa bytes) 000246[1] 1768008579.416503 mapwrite 0x2f8000 thru 0x2fec48 (0x6c49 bytes) 000247[1] 1768008579.496417 mapwrite 0x5d9000 thru 0x5e5718 (0xc719 bytes) 000249[1] 1768008579.567771 trunc from 0x5e5719 to 0x84c5de (0x266ec6 bytes) 000254[1] 1768008579.632779 write_OD 0x095000 thru 0x095fff (0x1000 bytes) 000257[0] 1768008579.688652 write 0x2c7000 thru 0x2cc74d (0x574e bytes) 000258[0] 1768008579.709575 mapread 0x44c000 thru 0x455d38 (0x9d39 bytes) 000260[1] 1768008579.746056 trunc from 0x84c5df to 0x94e50f (0x101f31 bytes) 000261[1] 1768008579.781493 write_OD 0x6d8000 thru 0x6e0fff (0x9000 bytes) 000262[0] 1768008579.809907 write 0x89c000 thru 0x8a4f1c (0x8f1d bytes) 000266[0] 1768008579.816456 mapwrite 0x147000 thru 0x15075c (0x975d bytes) 000267[0] 1768008579.857735 mapread 0x427000 thru 0x43603f (0xf040 bytes) 000273[1] 1768008579.894841 mapwrite 0x043000 thru 0x04f0d7 (0xc0d8 bytes) 000274[1] 1768008579.941473 write_OD 0x556000 thru 0x557fff (0x2000 bytes) 000275[1] 1768008579.962127 write_OD 0x2d0000 thru 0x2d5fff (0x6000 bytes) 000277[0] 1768008579.988962 trunc from 0x94e510 to 0x423d47 (0x52a7c8 bytes) 000278[1] 1768008580.002382 write_OD 0x5da000 thru 0x5dbfff (0x2000 bytes) 000280[1] 1768008580.034574 write_OD 0x46b000 thru 0x475fff (0xb000 bytes) 000281[1] 1768008580.072886 trunc from 0x5dc000 to 0x9ad635 (0x3d1636 bytes) 000283[1] 1768008580.116753 mapwrite 0x878000 thru 0x885157 (0xd158 bytes) 000284[0] 1768008580.161979 write 0x31d000 thru 0x3219f3 (0x49f4 bytes) 000287[1] 1768008580.206306 trunc from 0x9ad636 to 0x79aa75 (0x212bc0 bytes) 000288[1] 1768008580.225058 mapwrite 0x458000 thru 0x467602 (0xf603 bytes) 000290[1] 1768008580.269973 mapwrite 0x467000 thru 0x470067 (0x9068 bytes) 000292[0] 1768008580.303586 mapread 0x3b6000 thru 0x3bb6f8 (0x56f9 bytes) 000294[1] 1768008580.335554 read_OD 0x50f000 thru 0x518fff (0xa000 bytes) 000295[0] 1768008580.352999 write 0x20a000 thru 0x20c027 (0x2028 bytes) 000298[1] 1768008580.390847 trunc from 0x79aa76 to 0x40dcf7 (0x38cd7e bytes) 000299[0] 1768008580.416025 read 0x110000 thru 0x112a5c (0x2a5d bytes) 000302[0] 1768008580.434740 write 0x70a000 thru 0x713488 (0x9489 bytes) 000305[1] 1768008580.453306 mapread 0x094000 thru 0x0a2be4 (0xebe5 bytes) 000307[1] 1768008580.474851 trunc from 0x713489 to 0x17a6ca (0x598dbe bytes) 000308[1] 1768008580.508197 mapread 0x15f000 thru 0x1603ab (0x13ac bytes) 000310[1] 1768008580.523656 write_OD 0x6d0000 thru 0x6dcfff (0xd000 bytes) 000313[0] 1768008580.557881 mapwrite 0x5ed000 thru 0x5f2aa6 (0x5aa7 bytes) 000314[0] 1768008580.606594 mapwrite 0x190000 thru 0x199975 (0x9976 bytes) 000315[0] 1768008580.663616 mapread 0x0bb000 thru 0x0c6af1 (0xbaf2 bytes) 000316[0] 1768008580.671912 write 0x434000 thru 0x4345b3 (0x5b4 bytes) 000319[1] 1768008580.692847 trunc from 0x6dd000 to 0x542978 (0x19a687 bytes) 000323[1] 1768008580.725620 read_OD 0x1ec000 thru 0x1f9fff (0xe000 bytes) 000324[1] 1768008580.734737 write_OD 0x975000 thru 0x978fff (0x4000 bytes) 000327[0] 1768008580.762724 mapwrite 0x176000 thru 0x1831be (0xd1bf bytes) 000329[0] 1768008580.827481 mapread 0x610000 thru 0x611a86 (0x1a87 bytes) 000331[0] 1768008580.843469 mapwrite 0x70b000 thru 0x70d49b (0x249c bytes) 000334[1] 1768008580.901403 mapwrite 0x4b9000 thru 0x4b9a26 (0xa27 bytes) 000335[0] 1768008580.938479 read 0x844000 thru 0x847fad (0x3fae bytes) 000336[0] 1768008580.947985 mapwrite 0x436000 thru 0x43bdd5 (0x5dd6 bytes) 000338[0] 1768008581.017404 read 0x88b000 thru 0x899656 (0xe657 bytes) 000339[1] 1768008581.032676 trunc from 0x979000 to 0x35c5a8 (0x61ca57 bytes) 000341[0] 1768008581.057271 trunc from 0x35c5a9 to 0x7f4bf0 (0x498648 bytes) 000342[1] 1768008581.071971 trunc from 0x7f4bf1 to 0x48f350 (0x3658a0 bytes) 000345[0] 1768008581.094314 mapwrite 0x3fd000 thru 0x4058ad (0x88ae bytes) 000346[0] 1768008581.147455 mapread 0x283000 thru 0x289bc9 (0x6bca bytes) 000349[1] 1768008581.202935 read_OD 0x051000 thru 0x057fff (0x7000 bytes) 000354[0] 1768008581.226458 write 0x292000 thru 0x292986 (0x987 bytes) 000355[0] 1768008581.237347 trunc from 0x48f351 to 0x79e2cd (0x30ef7d bytes) 000356[1] 1768008581.252103 trunc from 0x79e2ce to 0x79eee4 (0xc17 bytes) 000357[1] 1768008581.277337 write_OD 0x412000 thru 0x41ffff (0xe000 bytes) 000361[0] 1768008581.322403 mapread 0x4e6000 thru 0x4e72a3 (0x12a4 bytes) 000362[1] 1768008581.335657 write_OD 0x3aa000 thru 0x3b4fff (0xb000 bytes) 000364[0] 1768008581.369708 mapread 0x776000 thru 0x779494 (0x3495 bytes) 000369[0] 1768008581.391173 mapread 0x482000 thru 0x48b0a6 (0x90a7 bytes) 000371[1] 1768008581.418357 mapread 0x33b000 thru 0x341307 (0x6308 bytes) 000372[0] 1768008581.426646 write 0x8b2000 thru 0x8b2c97 (0xc98 bytes) 000373[0] 1768008581.440024 write 0x66a000 thru 0x673a2a (0x9a2b bytes) 000375[0] 1768008581.453658 write 0x924000 thru 0x9296a6 (0x56a7 bytes) 000376[1] 1768008581.462284 trunc from 0x9296a7 to 0x9b02cc (0x86c26 bytes) 000377[1] 1768008581.489007 mapwrite 0x7ae000 thru 0x7b99d6 (0xb9d7 bytes) 000379[0] 1768008581.555762 trunc from 0x9b02cd to 0x121dfd (0x88e4cf bytes) 000383[0] 1768008581.588374 trunc from 0x121dfe to 0x88bf7f (0x76a182 bytes) 000384[0] 1768008581.628573 write 0x076000 thru 0x081437 (0xb438 bytes) 000385[1] 1768008581.656306 write_OD 0x0d9000 thru 0x0e1fff (0x9000 bytes) 000386[0] 1768008581.720759 mapread 0x6f2000 thru 0x6f4f88 (0x2f89 bytes) 000387[1] 1768008581.749427 write_OD 0x252000 thru 0x254fff (0x3000 bytes) 000388[1] 1768008581.786012 read_OD 0x2b5000 thru 0x2befff (0xa000 bytes) 000390[1] 1768008581.795787 trunc from 0x88bf80 to 0x5e5385 (0x2a6bfa bytes) 000395[1] 1768008581.882345 write_OD 0x88e000 thru 0x895fff (0x8000 bytes) 000396[0] 1768008581.918748 write 0x826000 thru 0x8288ec (0x28ed bytes) 000398[0] 1768008581.935942 write 0x156000 thru 0x15ba80 (0x5a81 bytes) 000399[1] 1768008581.939825 mapread 0x4b1000 thru 0x4b32c7 (0x22c8 bytes) 000401[0] 1768008581.965911 mapread 0x2ee000 thru 0x2fc2da (0xe2db bytes) 000404[0] 1768008581.978346 mapread 0x11b000 thru 0x124032 (0x9033 bytes) 000407[0] 1768008581.997134 mapwrite 0x613000 thru 0x61efa5 (0xbfa6 bytes) 000408[1] 1768008582.051734 read_OD 0x7d9000 thru 0x7e3fff (0xb000 bytes) 000410[0] 1768008582.074395 mapread 0x346000 thru 0x34952e (0x352f bytes) 000412[1] 1768008582.104991 read_OD 0x4b6000 thru 0x4b6fff (0x1000 bytes) 000419[0] 1768008582.113004 trunc from 0x896000 to 0x7b4c66 (0xe1399 bytes) 000421[0] 1768008582.163180 trunc from 0x7b4c67 to 0x3be31b (0x3f694b bytes) 000423[0] 1768008582.201290 mapread 0x3bd000 thru 0x3bd7a3 (0x7a4 bytes) 000424[0] 1768008582.234977 write 0x04c000 thru 0x05bc6f (0xfc70 bytes) 000426[1] 1768008582.251513 write_OD 0x20c000 thru 0x219fff (0xe000 bytes) 000428[1] 1768008582.290681 mapwrite 0x913000 thru 0x91b542 (0x8543 bytes) 000429[1] 1768008582.350967 mapwrite 0x336000 thru 0x34153b (0xb53c bytes) 000433[0] 1768008582.437284 trunc from 0x91b543 to 0x5897a0 (0x391da2 bytes) 000434[1] 1768008582.482012 write_OD 0x2cd000 thru 0x2dbfff (0xf000 bytes) 000435[1] 1768008582.530190 read_OD 0x006000 thru 0x013fff (0xe000 bytes) 000436[1] 1768008582.558640 trunc from 0x5897a1 to 0x3af1b1 (0x1da5ef bytes) 000438[0] 1768008582.603543 read 0x061000 thru 0x06f6fd (0xe6fe bytes) 000440[0] 1768008582.625061 read 0x1e4000 thru 0x1ecc66 (0x8c67 bytes) 000442[1] 1768008582.674021 trunc from 0x3af1b2 to 0x344324 (0x6ae8d bytes) 000444[1] 1768008582.726522 trunc from 0x344325 to 0x3dd01b (0x98cf7 bytes) 000446[1] 1768008582.735692 mapread 0x0eb000 thru 0x0f0f53 (0x5f54 bytes) 000448[0] 1768008582.758663 mapread 0x203000 thru 0x203f20 (0xf21 bytes) 000449[0] 1768008582.777314 write 0x051000 thru 0x05fb40 (0xeb41 bytes) 000450[0] 1768008582.787907 mapwrite 0x929000 thru 0x930fa1 (0x7fa2 bytes) 000451[0] 1768008582.897934 mapread 0x737000 thru 0x73e47f (0x7480 bytes) 000453[1] 1768008582.898891 write_OD 0x917000 thru 0x923fff (0xd000 bytes) 000455[0] 1768008582.939772 mapread 0x69c000 thru 0x69e9a5 (0x29a6 bytes) 000457[0] 1768008582.966709 write 0x2fb000 thru 0x2ffa7e (0x4a7f bytes) 000458[0] 1768008583.002480 mapwrite 0x686000 thru 0x689e15 (0x3e16 bytes) 000464[1] 1768008583.088747 mapwrite 0x05d000 thru 0x063ed7 (0x6ed8 bytes) 000465[1] 1768008583.147012 write_OD 0x23a000 thru 0x247fff (0xe000 bytes) 000466[1] 1768008583.175088 mapwrite 0x8b5000 thru 0x8b6e8a (0x1e8b bytes) 000467[0] 1768008583.231932 mapread 0x2c8000 thru 0x2cdabe (0x5abf bytes) 000468[1] 1768008583.295573 trunc from 0x930fa2 to 0x5a4522 (0x38ca7f bytes) 000469[0] 1768008583.344619 mapwrite 0x652000 thru 0x658297 (0x6298 bytes) 000472[0] 1768008583.468158 read 0x312000 thru 0x314c7d (0x2c7e bytes) 000476[0] 1768008583.512301 mapread 0x568000 thru 0x569502 (0x1503 bytes) 000477[1] 1768008583.548582 write_OD 0x858000 thru 0x866fff (0xf000 bytes) truncating to largest ever: 0x9d0cab 000478[1] 1768008583.590775 trunc from 0x867000 to 0x9d0caa (0x169cab bytes) 000482[0] 1768008583.659404 trunc from 0x9d0cab to 0x59f979 (0x431331 bytes) 000485[1] 1768008583.695348 mapwrite 0x51c000 thru 0x529f14 (0xdf15 bytes) 000489[1] 1768008583.775720 read_OD 0x532000 thru 0x538fff (0x7000 bytes) 000491[0] 1768008583.802480 mapwrite 0x0e4000 thru 0x0e5898 (0x1899 bytes) 000492[0] 1768008583.858452 trunc from 0x59f97a to 0x81feed (0x280574 bytes) 000496[1] 1768008583.901826 mapwrite 0x48e000 thru 0x496c7c (0x8c7d bytes) 000497[0] 1768008583.957968 write 0x7c5000 thru 0x7d0557 (0xb558 bytes) All operations completed A-OK! PASS 16a (25s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 20:29:46 (1768008586) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.167453 s, 50.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.158208 s, 53.0 MB/s Chance of close/open is 1 in 50 Seed set to 8588 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x914289 000001[1] 1768008588.387018 trunc from 00000000 to 0x914288 (0x914289 bytes) 000003[1] 1768008588.432876 mapwrite 0x0c7013 thru 0x0c7fc0 (0xfae bytes) 000004[1] 1768008588.491762 write 0x7bc370 thru 0x7bd487 (0x1118 bytes) 000005[1] 1768008588.505645 read 0x89842c thru 0x898eb9 (0xa8e bytes) 000006[1] 1768008588.528708 mapwrite 0x6b8ec6 thru 0x6ba9c1 (0x1afc bytes) 000007[0] 1768008588.606639 trunc from 0x914289 to 0x85590d (0xbe97b bytes) 000009[0] 1768008588.677596 trunc from 0x85590e to 0x1de084 (0x677889 bytes) 000010[0] 1768008588.719044 mapwrite 0x1ea665 thru 0x1eb6da (0x1076 bytes) 000011[0] 1768008588.772845 write 0x203bfc thru 0x204828 (0xc2d bytes) 000014[1] 1768008588.793492 trunc from 0x204829 to 0x57d167 (0x37893f bytes) 000017[1] 1768008588.824560 mapread 0x15f8d0 thru 0x1613d0 (0x1b01 bytes) 000020[0] 1768008588.871199 trunc from 0x57d168 to 0x091137 (0x4ec030 bytes) 000021[0] 1768008588.918314 write 0x89c67a thru 0x89d202 (0xb89 bytes) 000022[0] 1768008588.926373 mapread 0x71fba2 thru 0x7214a0 (0x18ff bytes) 000024[0] 1768008588.988686 mapwrite 0x2a33ef thru 0x2a45c8 (0x11da bytes) 000025[1] 1768008589.019359 trunc from 0x89d203 to 0x1a6cad (0x6f6555 bytes) 000030[1] 1768008589.122823 trunc from 0x1a6cae to 0x3ac98e (0x205ce1 bytes) 000033[1] 1768008589.175922 write 0x42acf5 thru 0x42c188 (0x1494 bytes) 000034[1] 1768008589.188268 read 0x0616cb thru 0x061f25 (0x85b bytes) 000037[1] 1768008589.193137 mapread 0x00d872 thru 0x00e6b9 (0xe48 bytes) 000038[1] 1768008589.201788 trunc from 0x42c189 to 0x12c518 (0x2ffc70 bytes) 000039[0] 1768008589.214163 mapread 0x102c48 thru 0x1049da (0x1d93 bytes) 000040[0] 1768008589.245393 read 0x038f23 thru 0x039cf1 (0xdcf bytes) 000044[0] 1768008589.287721 read 0x0157e5 thru 0x016f34 (0x1750 bytes) 000048[1] 1768008589.357901 mapread 0x07f03c thru 0x07f385 (0x34a bytes) 000050[0] 1768008589.381939 trunc from 0x12c519 to 0x8ca513 (0x79dffb bytes) 000054[0] 1768008589.463059 mapread 0x7e44b4 thru 0x7e59fd (0x154a bytes) 000055[1] 1768008589.586061 write 0x872a46 thru 0x8741ce (0x1789 bytes) 000057[1] 1768008589.633892 trunc from 0x8ca514 to 0x1c055f (0x709fb4 bytes) 000058[1] 1768008589.663886 mapread 0x12235e thru 0x122bdc (0x87f bytes) 000059[0] 1768008589.685045 read 0x0f5620 thru 0x0f6bfd (0x15de bytes) 000060[1] 1768008589.722713 mapread 0x0b3172 thru 0x0b4cea (0x1b79 bytes) 000061[0] 1768008589.735064 mapread 0x0c38b6 thru 0x0c4388 (0xad3 bytes) 000063[1] 1768008589.742202 read 0x0e52c5 thru 0x0e6fed (0x1d29 bytes) 000064[1] 1768008589.785070 read 0x120557 thru 0x120ff4 (0xa9e bytes) 000065[1] 1768008589.786680 trunc from 0x1c0560 to 0x36d6f0 (0x1ad191 bytes) 000066[1] 1768008589.832596 mapwrite 0x65e718 thru 0x65eabe (0x3a7 bytes) 000067[0] 1768008589.903914 mapwrite 0x789e30 thru 0x78b590 (0x1761 bytes) 000068[0] 1768008590.063775 mapread 0x130014 thru 0x131943 (0x1930 bytes) 000069[1] 1768008590.106203 mapread 0x2b0fa2 thru 0x2b1dc2 (0xe21 bytes) 000070[0] 1768008590.147780 trunc from 0x78b591 to 0x880f7d (0xf59ed bytes) 000071[0] 1768008590.213847 trunc from 0x880f7e to 0x86a939 (0x16644 bytes) 000072[0] 1768008590.227170 mapwrite 0x2208d7 thru 0x22115e (0x888 bytes) 000077[1] 1768008590.312073 write 0x18b8aa thru 0x18cf39 (0x1690 bytes) 000078[0] 1768008590.341277 write 0x3986fe thru 0x39a183 (0x1a86 bytes) 000079[1] 1768008590.358319 mapwrite 0x9fe5e7 thru 0x9ffc64 (0x167e bytes) 000081[0] 1768008590.537339 write 0x493b83 thru 0x4941c9 (0x647 bytes) 000082[1] 1768008590.572023 mapwrite 0x66dcc3 thru 0x66e66e (0x9ac bytes) 000086[1] 1768008590.699158 read 0x996a49 thru 0x997d53 (0x130b bytes) 000088[1] 1768008590.727168 write 0x6f2989 thru 0x6f3a36 (0x10ae bytes) 000089[0] 1768008590.751627 trunc from 0x9ffc65 to 0x87f2f0 (0x180974 bytes) 000090[0] 1768008590.779807 mapwrite 0x9f5469 thru 0x9f5e33 (0x9cb bytes) 000091[1] 1768008590.873903 mapwrite 0x9840ea thru 0x985314 (0x122b bytes) 000093[1] 1768008590.961372 write 0x3b413e thru 0x3b4dbe (0xc81 bytes) 000094[0] 1768008590.978592 mapwrite 0x5b1174 thru 0x5b1998 (0x825 bytes) 000097[1] 1768008591.090269 trunc from 0x9f5e34 to 0x24bd2a (0x7aa109 bytes) 000098[0] 1768008591.127666 trunc from 0x24bd2b to 0x272f93 (0x27269 bytes) 000106[0] 1768008591.267913 mapwrite 0x12c816 thru 0x12db98 (0x1383 bytes) 000109[1] 1768008591.331306 mapwrite 0x4b86c3 thru 0x4b9b24 (0x1462 bytes) 000112[0] 1768008591.391599 mapread 0x2009f0 thru 0x201da0 (0x13b1 bytes) 000121[0] 1768008591.462015 trunc from 0x4b9b25 to 0x14a920 (0x36f204 bytes) 000122[0] 1768008591.498556 read 0x007aa6 thru 0x008e6b (0x13c6 bytes) 000124[0] 1768008591.530581 read 0x109d77 thru 0x10a1a1 (0x42b bytes) 000125[0] 1768008591.539835 read 0x034984 thru 0x0361ab (0x1828 bytes) 000126[0] 1768008591.540336 read 0x0a6092 thru 0x0a7c53 (0x1bc2 bytes) 000127[1] 1768008591.540711 trunc from 0x14a921 to 0x639b32 (0x4ef212 bytes) 000128[1] 1768008591.577598 trunc from 0x639b33 to 0x14571e (0x4f4414 bytes) 000130[0] 1768008591.611928 read 0x059f3e thru 0x05ab74 (0xc37 bytes) 000131[0] 1768008591.649143 write 0x110c5f thru 0x111558 (0x8fa bytes) 000132[0] 1768008591.689380 trunc from 0x14571f to 0x3d294e (0x28d230 bytes) 000133[0] 1768008591.710874 mapwrite 0x8833b0 thru 0x884ed2 (0x1b23 bytes) 000134[0] 1768008591.839263 read 0x5ad98c thru 0x5ae375 (0x9ea bytes) truncating to largest ever: 0x92d547 000135[1] 1768008591.840002 trunc from 0x884ed3 to 0x92d546 (0xa8674 bytes) 000137[1] 1768008591.881148 read 0x098505 thru 0x098e86 (0x982 bytes) 000139[0] 1768008591.899772 read 0x106719 thru 0x10718d (0xa75 bytes) 000142[0] 1768008591.933832 mapwrite 0x199009 thru 0x19a0d6 (0x10ce bytes) 000147[0] 1768008592.120715 write 0x7e3223 thru 0x7e447d (0x125b bytes) 000149[0] 1768008592.150183 mapwrite 0x00c1a0 thru 0x00cb7b (0x9dc bytes) 000150[1] 1768008592.193095 mapwrite 0x98c648 thru 0x98cb52 (0x50b bytes) 000156[0] 1768008592.344134 mapwrite 0x56e4c3 thru 0x56f10e (0xc4c bytes) 000157[0] 1768008592.424090 mapread 0x530802 thru 0x531c8b (0x148a bytes) 000158[0] 1768008592.453793 trunc from 0x98cb53 to 0x087689 (0x9054c9 bytes) 000159[1] 1768008592.484452 read 0x0835f6 thru 0x083c7d (0x688 bytes) 000162[0] 1768008592.504689 mapwrite 0x3162ef thru 0x316cc5 (0x9d7 bytes) 000164[0] 1768008592.605672 read 0x180729 thru 0x1809a7 (0x27f bytes) 000165[1] 1768008592.646126 write 0x20e2f9 thru 0x20fde9 (0x1af1 bytes) 000169[1] 1768008592.721919 read 0x155d0a thru 0x1568d1 (0xbc8 bytes) 000170[0] 1768008592.802510 read 0x2c9ea4 thru 0x2cb046 (0x11a3 bytes) 000171[0] 1768008592.884530 read 0x158724 thru 0x159331 (0xc0e bytes) 000172[1] 1768008592.885519 mapread 0x2c5156 thru 0x2c678b (0x1636 bytes) 000173[1] 1768008592.914316 read 0x28914c thru 0x28960c (0x4c1 bytes) 000174[1] 1768008592.961543 mapread 0x16255a thru 0x163a6a (0x1511 bytes) 000177[1] 1768008592.981321 read 0x13faa1 thru 0x141164 (0x16c4 bytes) 000181[0] 1768008593.002187 read 0x1e7e68 thru 0x1e8072 (0x20b bytes) 000187[1] 1768008593.060510 read 0x2e0351 thru 0x2e03df (0x8f bytes) 000188[0] 1768008593.071364 trunc from 0x316cc6 to 0x09b316 (0x27b9af bytes) 000190[0] 1768008593.141685 mapwrite 0x2ad11b thru 0x2ad63e (0x524 bytes) 000192[0] 1768008593.348744 mapread 0x060799 thru 0x06091f (0x187 bytes) 000195[1] 1768008593.412843 read 0x2ac849 thru 0x2ad63e (0xdf6 bytes) 000197[1] 1768008593.498531 mapread 0x166c7a thru 0x1672f8 (0x67f bytes) 000198[1] 1768008593.513442 mapwrite 0x8813c5 thru 0x88278d (0x13c9 bytes) 000200[0] 1768008593.674178 mapread 0x286eac thru 0x2886d4 (0x1829 bytes) 000202[1] 1768008593.747742 trunc from 0x88278e to 0x075110 (0x80d67d bytes) 000203[0] 1768008593.773362 mapread 0x012d2d thru 0x014793 (0x1a67 bytes) 000204[0] 1768008593.823417 trunc from 0x075111 to 0x7f04b8 (0x77b3a8 bytes) 000205[1] 1768008593.870128 trunc from 0x7f04b9 to 0x8da3cf (0xe9f17 bytes) 000206[1] 1768008593.907804 mapread 0x1fc9f4 thru 0x1fd845 (0xe52 bytes) 000208[0] 1768008593.934608 mapread 0x057f68 thru 0x0590ff (0x1198 bytes) 000212[1] 1768008594.022377 write 0x00b816 thru 0x00c0d5 (0x8c0 bytes) 000213[0] 1768008594.046767 write 0x6d0cdf thru 0x6d297e (0x1ca0 bytes) 000219[1] 1768008594.221666 write 0x783d78 thru 0x784254 (0x4dd bytes) 000220[1] 1768008594.250321 trunc from 0x8da3d0 to 0x2f02b2 (0x5ea11d bytes) 000221[1] 1768008594.295398 mapwrite 0x09e5dd thru 0x09f2ef (0xd13 bytes) 000222[0] 1768008594.367731 write 0x4651a3 thru 0x4656b6 (0x514 bytes) 000223[0] 1768008594.404012 write 0x8c9b40 thru 0x8cad3b (0x11fc bytes) 000226[0] 1768008594.441968 mapread 0x5fad87 thru 0x5fba71 (0xceb bytes) 000227[0] 1768008594.473415 write 0x63b936 thru 0x63bbb7 (0x282 bytes) 000228[0] 1768008594.495397 trunc from 0x8cad3c to 0x80584e (0xc54ed bytes) 000230[0] 1768008594.543375 write 0x16eacc thru 0x170169 (0x169e bytes) 000231[0] 1768008594.586654 read 0x60d51a thru 0x60e3bb (0xea2 bytes) 000234[0] 1768008594.613513 mapwrite 0x29f7e4 thru 0x29f9ea (0x207 bytes) 000236[1] 1768008594.695417 trunc from 0x80584f to 0x03b632 (0x7ca21c bytes) 000237[0] 1768008594.732297 mapread 0x02a3f4 thru 0x02bbc8 (0x17d5 bytes) 000239[0] 1768008594.767151 mapread 0x009b8b thru 0x009e19 (0x28f bytes) 000241[0] 1768008594.782643 trunc from 0x03b633 to 0x33a88f (0x2ff25d bytes) 000242[1] 1768008594.820151 write 0x035dd5 thru 0x036427 (0x653 bytes) 000243[1] 1768008594.846062 write 0x9a3264 thru 0x9a4427 (0x11c4 bytes) 000245[1] 1768008594.881846 read 0x6bc1c2 thru 0x6bc7fc (0x63b bytes) 000247[0] 1768008594.900733 write 0x1cd65d thru 0x1cdb75 (0x519 bytes) 000248[1] 1768008594.928596 mapwrite 0x31ec22 thru 0x31ffd3 (0x13b2 bytes) 000249[1] 1768008594.979432 read 0x0c7b71 thru 0x0c8ae2 (0xf72 bytes) 000255[1] 1768008595.003473 mapread 0x39db1c thru 0x39e151 (0x636 bytes) 000257[1] 1768008595.020348 read 0x800d8f thru 0x802b06 (0x1d78 bytes) 000262[0] 1768008595.055783 mapread 0x354e3a thru 0x356b3c (0x1d03 bytes) 000264[1] 1768008595.097896 trunc from 0x9a4428 to 0x326e8e (0x67d599 bytes) 000268[0] 1768008595.160594 read 0x18dd51 thru 0x18f946 (0x1bf6 bytes) 000269[0] 1768008595.210793 read 0x315907 thru 0x315b3c (0x236 bytes) 000271[1] 1768008595.221053 mapwrite 0x6d39bd thru 0x6d3ade (0x122 bytes) 000272[1] 1768008595.319121 read 0x04cc01 thru 0x04e53f (0x193f bytes) 000276[0] 1768008595.366167 write 0x5b571b thru 0x5b6352 (0xc38 bytes) 000277[1] 1768008595.400578 trunc from 0x6d3adf to 0x1ebb58 (0x4e7f86 bytes) 000278[0] 1768008595.444701 mapwrite 0x6a0606 thru 0x6a0712 (0x10d bytes) 000279[0] 1768008595.552139 trunc from 0x6a0713 to 0x24ae37 (0x4558db bytes) 000280[1] 1768008595.579034 mapread 0x0cd6eb thru 0x0ce73e (0x1054 bytes) 000282[1] 1768008595.639150 read 0x08137a thru 0x081ebc (0xb43 bytes) 000285[0] 1768008595.692972 read 0x171556 thru 0x1722b7 (0xd62 bytes) 000286[1] 1768008595.722662 write 0x2d0ef8 thru 0x2d29ce (0x1ad7 bytes) 000287[1] 1768008595.750801 write 0x9494f2 thru 0x94aecf (0x19de bytes) 000288[0] 1768008595.769827 mapread 0x13f300 thru 0x13f8a4 (0x5a5 bytes) 000291[1] 1768008595.842743 mapwrite 0x6e4396 thru 0x6e4d8c (0x9f7 bytes) 000292[1] 1768008595.912869 write 0x2d81be thru 0x2d9507 (0x134a bytes) 000293[1] 1768008595.934057 trunc from 0x94aed0 to 0x3c1386 (0x589b49 bytes) 000294[1] 1768008595.959656 mapread 0x115c57 thru 0x11761a (0x19c4 bytes) 000295[0] 1768008596.006291 mapwrite 0x1204cb thru 0x121242 (0xd78 bytes) 000296[0] 1768008596.083525 mapwrite 0x62f94f thru 0x63175d (0x1e0f bytes) 000299[0] 1768008596.164097 write 0x9b35ea thru 0x9b3e94 (0x8ab bytes) 000301[1] 1768008596.198702 mapread 0x72600c thru 0x72711b (0x1110 bytes) truncating to largest ever: 0x954f36 000302[1] 1768008596.249130 trunc from 0x9b3e95 to 0x954f35 (0x5ef5f bytes) 000306[0] 1768008596.320166 mapwrite 0x2dd053 thru 0x2dd5a2 (0x550 bytes) 000311[1] 1768008596.412994 trunc from 0x954f36 to 0x1c181d (0x793718 bytes) 000312[1] 1768008596.459911 write 0x62cc88 thru 0x62d59f (0x918 bytes) 000313[0] 1768008596.474087 mapwrite 0x6b01b6 thru 0x6b1475 (0x12c0 bytes) 000314[1] 1768008596.566766 write 0x6431d8 thru 0x644349 (0x1172 bytes) 000315[0] 1768008596.603253 trunc from 0x6b1476 to 0x6d48f5 (0x23480 bytes) 000316[1] 1768008596.645859 trunc from 0x6d48f6 to 0x4d076b (0x20418a bytes) 000317[0] 1768008596.700152 read 0x13716b thru 0x13852d (0x13c3 bytes) 000318[0] 1768008596.714122 mapread 0x44011c thru 0x441c51 (0x1b36 bytes) 000319[1] 1768008596.744324 read 0x3d687b thru 0x3d79b7 (0x113d bytes) 000320[1] 1768008596.748122 mapread 0x086a47 thru 0x0872c3 (0x87d bytes) 000322[0] 1768008596.756980 write 0x16023c thru 0x1618f1 (0x16b6 bytes) 000327[0] 1768008596.802415 mapwrite 0x06521a thru 0x0655c4 (0x3ab bytes) 000328[0] 1768008596.828363 mapread 0x3cf4f0 thru 0x3cf551 (0x62 bytes) 000329[0] 1768008596.831854 read 0x378942 thru 0x379b90 (0x124f bytes) 000330[1] 1768008596.838708 trunc from 0x4d076c to 0x751429 (0x280cbe bytes) 000335[1] 1768008596.896535 trunc from 0x75142a to 0x6194c2 (0x137f67 bytes) 000337[0] 1768008596.922854 read 0x0ac1e4 thru 0x0acf91 (0xdae bytes) 000338[1] 1768008596.942508 mapread 0x07a28e thru 0x07b977 (0x16ea bytes) 000340[0] 1768008596.953952 write 0x716c89 thru 0x717ba8 (0xf20 bytes) 000343[1] 1768008596.979112 read 0x45e270 thru 0x45fa40 (0x17d1 bytes) 000345[1] 1768008596.992157 mapwrite 0x42d150 thru 0x42e1a5 (0x1056 bytes) 000346[1] 1768008597.027986 trunc from 0x717ba9 to 0x4b2c95 (0x264f13 bytes) 000350[0] 1768008597.062711 write 0x954130 thru 0x9543b5 (0x286 bytes) 000352[0] 1768008597.081340 write 0x05db7e thru 0x05e8b6 (0xd39 bytes) 000354[0] 1768008597.100845 mapwrite 0x8cc55e thru 0x8ce16a (0x1c0d bytes) 000356[0] 1768008597.162574 write 0x1ce485 thru 0x1cea87 (0x603 bytes) 000358[1] 1768008597.197038 mapwrite 0x1e1869 thru 0x1e1fae (0x746 bytes) 000359[1] 1768008597.236385 write 0x06e3ec thru 0x06efa8 (0xbbd bytes) 000360[0] 1768008597.246652 mapwrite 0x56bec1 thru 0x56cc16 (0xd56 bytes) 000361[0] 1768008597.306290 write 0x164eb9 thru 0x165318 (0x460 bytes) 000363[0] 1768008597.340705 read 0x09865b thru 0x09a52d (0x1ed3 bytes) 000364[0] 1768008597.343643 mapwrite 0x78fa29 thru 0x790266 (0x83e bytes) 000372[0] 1768008597.430379 trunc from 0x9543b6 to 0x23c914 (0x717aa1 bytes) 000373[0] 1768008597.439753 mapwrite 0x81bbbf thru 0x81bc2c (0x6e bytes) 000375[1] 1768008597.482662 write 0x3c6f92 thru 0x3c7c54 (0xcc3 bytes) 000377[1] 1768008597.504349 read 0x6680cd thru 0x668d2f (0xc63 bytes) 000380[0] 1768008597.511366 read 0x603843 thru 0x6055e0 (0x1d9e bytes) 000381[0] 1768008597.524346 mapwrite 0x2425d5 thru 0x2439f6 (0x1422 bytes) 000384[0] 1768008597.578546 mapwrite 0x41370d thru 0x4156d3 (0x1fc7 bytes) 000385[0] 1768008597.618881 read 0x7bd77a thru 0x7bf31f (0x1ba6 bytes) 000387[0] 1768008597.662002 mapwrite 0x101e7e thru 0x1031e4 (0x1367 bytes) 000388[1] 1768008597.705341 trunc from 0x81bc2d to 0x4527f3 (0x3c9439 bytes) 000393[0] 1768008597.764062 read 0x04a396 thru 0x04c248 (0x1eb3 bytes) 000394[0] 1768008597.781813 write 0x2d524e thru 0x2d70d6 (0x1e89 bytes) 000395[1] 1768008597.822662 write 0x81c7fc thru 0x81e11c (0x1921 bytes) 000396[1] 1768008597.836328 mapwrite 0x6ea9ac thru 0x6eb0c6 (0x71b bytes) 000398[0] 1768008597.885069 read 0x2b37f5 thru 0x2b4219 (0xa25 bytes) 000399[0] 1768008597.905208 mapread 0x597bd4 thru 0x598119 (0x546 bytes) 000400[1] 1768008597.922560 mapread 0x333b23 thru 0x333fe3 (0x4c1 bytes) 000403[1] 1768008597.925812 trunc from 0x81e11d to 0x5ecfa1 (0x23117b bytes) 000404[1] 1768008597.958966 mapread 0x5d194b thru 0x5d2d27 (0x13dd bytes) 000408[0] 1768008598.022673 mapwrite 0x772387 thru 0x772d88 (0xa02 bytes) 000409[0] 1768008598.108594 read 0x7202f5 thru 0x7205cf (0x2db bytes) 000410[0] 1768008598.123456 mapread 0x449912 thru 0x44b188 (0x1877 bytes) 000411[1] 1768008598.146100 trunc from 0x772d89 to 0x6c219f (0xb0be9 bytes) 000414[0] 1768008598.181151 read 0x64e065 thru 0x64f38e (0x132a bytes) 000417[1] 1768008598.209843 mapwrite 0x7a7397 thru 0x7a8f58 (0x1bc2 bytes) 000418[0] 1768008598.255587 read 0x4da4ad thru 0x4da6ad (0x201 bytes) 000424[1] 1768008598.319299 write 0x1b5d87 thru 0x1b6565 (0x7df bytes) 000426[1] 1768008598.338049 write 0x4395e9 thru 0x43b342 (0x1d5a bytes) 000429[1] 1768008598.383787 trunc from 0x7a8f59 to 0x7a351e (0x5a3a bytes) 000431[0] 1768008598.411804 trunc from 0x7a351f to 0x479a3d (0x329ae1 bytes) 000432[1] 1768008598.443588 read 0x05381e thru 0x054c53 (0x1436 bytes) 000435[0] 1768008598.494679 trunc from 0x479a3e to 0x6c180f (0x247dd2 bytes) 000436[1] 1768008598.558051 read 0x019ffb thru 0x01a028 (0x2e bytes) 000438[1] 1768008598.586552 write 0x2a7b6b thru 0x2a8f9d (0x1433 bytes) 000439[0] 1768008598.609728 trunc from 0x6c1810 to 0x2f1730 (0x3d00df bytes) 000441[0] 1768008598.646087 read 0x04e05a thru 0x04ee03 (0xdaa bytes) 000444[0] 1768008598.685495 write 0x9a1edf thru 0x9a226e (0x390 bytes) 000449[0] 1768008598.719565 mapread 0x87b6c0 thru 0x87ce6b (0x17ac bytes) 000454[1] 1768008598.873725 write 0x679030 thru 0x67971d (0x6ee bytes) 000455[1] 1768008598.920938 write 0x740199 thru 0x741116 (0xf7e bytes) 000456[0] 1768008598.936246 read 0x06a913 thru 0x06c123 (0x1811 bytes) 000459[1] 1768008598.968908 mapwrite 0x820c87 thru 0x822674 (0x19ee bytes) 000460[0] 1768008599.068357 read 0x8dedd5 thru 0x8df159 (0x385 bytes) 000461[0] 1768008599.083051 write 0x8cf71d thru 0x8cfbf4 (0x4d8 bytes) 000462[0] 1768008599.126130 mapread 0x3f5094 thru 0x3f6c49 (0x1bb6 bytes) 000466[0] 1768008599.211484 read 0x8234b1 thru 0x82429c (0xdec bytes) 000468[0] 1768008599.257822 mapwrite 0x2725ea thru 0x27298b (0x3a2 bytes) 000470[1] 1768008599.356427 trunc from 0x9a226f to 0x9450e8 (0x5d186 bytes) 000475[0] 1768008599.429681 write 0x91f4c9 thru 0x9211aa (0x1ce2 bytes) 000477[0] 1768008599.464925 trunc from 0x9450e9 to 0x3e8c7b (0x55c46d bytes) 000478[0] 1768008599.510357 mapread 0x01ed73 thru 0x01f256 (0x4e4 bytes) 000479[0] 1768008599.520439 write 0x7f7ccf thru 0x7f855e (0x890 bytes) 000480[0] 1768008599.537831 trunc from 0x7f855f to 0x5ec3bb (0x20c1a3 bytes) 000481[1] 1768008599.558392 trunc from 0x5ec3bc to 0x43382c (0x1b8b8f bytes) 000482[1] 1768008599.585612 read 0x0d5c8d thru 0x0d7356 (0x16ca bytes) 000484[0] 1768008599.590087 write 0x6ab31f thru 0x6ad2af (0x1f91 bytes) 000485[0] 1768008599.614587 read 0x2aacd9 thru 0x2aba8b (0xdb3 bytes) 000486[1] 1768008599.632554 read 0x67dd0a thru 0x67f18b (0x1482 bytes) 000489[1] 1768008599.663491 mapwrite 0x8ee991 thru 0x8ef13c (0x7ac bytes) 000490[0] 1768008599.750643 trunc from 0x8ef13d to 0x785f63 (0x1691d9 bytes) 000491[1] 1768008599.805012 trunc from 0x785f64 to 0x1316e5 (0x65487e bytes) 000492[0] 1768008599.860487 write 0x637994 thru 0x638e72 (0x14df bytes) 000493[1] 1768008599.882050 trunc from 0x638e73 to 0x785361 (0x14c4ef bytes) 000494[0] 1768008599.897299 mapread 0x5e9dcf thru 0x5ea90d (0xb3f bytes) 000498[0] 1768008599.944681 read 0x754f5c thru 0x756bfb (0x1ca0 bytes) All operations completed A-OK! PASS 16b (16s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 20:30:02 (1768008602) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 20:30:05 (1768008605) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.201394 s, 41.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.22637 s, 37.1 MB/s ldlm.namespaces.MGC192.168.203.146@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9aa147d3e800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9aa14a5b9800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9aa14a5b9800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9aa14a5b9800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18232 s, 20.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.71847 s, 61.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.28992 s, 81.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.32721 s, 31.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.13181 s, 25.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.74117 s, 60.2 MB/s PASS 16d (24s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 20:30:28 (1768008628) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.130338 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762613 s, 13.7 MB/s PASS 16e (4s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 20:30:32 (1768008632) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 20:30:56 (1768008656) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 20:31:19 (1768008679) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (4s) == sanityn test 16i: read after truncate file ============ 20:31:23 (1768008683) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 20:31:26 (1768008686) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.794982 s, 33.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.496259 s, 52.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.02068 s, 25.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.581881 s, 45.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.807438 s, 32.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.625457 s, 41.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.813637 s, 32.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.76658 s, 34.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.81853 s, 32.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.0111 s, 25.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.07397 s, 24.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.662107 s, 39.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.713613 s, 36.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.815361 s, 32.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.855992 s, 30.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.833798 s, 31.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.97009 s, 27.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11431 s, 23.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.15769 s, 22.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (14s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 20:31:40 (1768008700) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Jan 9 20:31:54 EST 2026 + fsx 2 done: rc=0 Fri Jan 9 20:31:54 EST 2026 + fsx 3 done: rc=0 Fri Jan 9 20:31:54 EST 2026 + fsx 4 done: rc=0 Fri Jan 9 20:31:54 EST 2026 + fsx 5 done: rc=0 Fri Jan 9 20:31:54 EST 2026 PASS 16k (19s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 20:31:59 (1768008719) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 20:32:05 (1768008725) mmap test1: basic mmap operation (PASS, 0.077453s) mmap test2: MAP_PRIVATE not write back (PASS, 0.046163s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.435s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1594s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.050161s) mmap test6: check mmap write/read content on two nodes (PASS, 0.043238s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.42293s) mmap test11: multi-thread shared mmap (PASS, 0.46505s) mmap test10: mtime not change for readonly mmap access (PASS, 1.045s) PASS 18 (16s) == sanityn test 19: test concurrent uncached read races ========================================================================= 20:32:21 (1768008741) oleg346-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 20:32:23 (1768008743) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 20:32:26 (1768008746) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 20:32:28 (1768008748) now is 1768008749 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 new atime is 1768008810 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 20:33:32 (1768008812) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.5K 386 490.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502242 386 501856 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234850 386 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 20:33:34 (1768008814) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 20:33:37 (1768008817) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 20:33:39 (1768008819) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 20:33:40 (1768008820) utime: good mknod times 1768008820 <= 1768008821 <= 1768008821 for /mnt/lustre/f26a utime: good mknod times 1768008820 <= 1768008821 <= 1768008821 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 20:33:44 (1768008824) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 20:33:48 (1768008828) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-09 20:33:49.000000000 -0500 Birth: 2026-01-09 20:33:49.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-01-09 20:33:49.000000000 -0500 Birth: 2026-01-09 20:33:49.000000000 -0500 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 20:33:52 (1768008832) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 20:33:57 (1768008837) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 20:34:00 (1768008840) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg346-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.146@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (12s) == sanityn test 31r: open-rename(replace) race =========== 20:34:12 (1768008852) -rw-r--r-- 1 root root 0 Jan 9 20:34 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 20:34:18 (1768008858) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42814 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 20:34:21 (1768008861) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 20:34:25 (1768008865) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 20:34:27 (1768008867) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 20:34:28 (1768008868) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 20:34:29 (1768008869) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 20:34:31 (1768008871) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid in IDLE state after 0 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid in FULL state after 0 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid in IDLE state after 0 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid in FULL state after 0 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa147d3e800.ost_server_uuid in IDLE state after 0 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa147d3e800.ost_server_uuid in IDLE state after 0 sec PASS 34 (95s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 20:36:06 (1768008966) fail_loc=0x80000317 Race attempt 0 Wait for 48865 48993 for 60 sec... - open/close 2263 (time 1768008978.84 total 10.00 last 226.23) total: 4000 open/close in 16.81 seconds: 238.02 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.146@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.146@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 20:37:11 (1768009031) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg346-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.489208 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.493987 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.488026 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.506138 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.476256 s, 110 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.464078 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.48596 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.467734 s, 112 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.436749 s, 120 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.420556 s, 125 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.43502 s, 121 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7443456) after(7494656) PASS 36 (240s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 20:41:11 (1768009271) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 - create 6367 (time 1768009283.26 total 10.00 last 636.60) total: 10000 create in 15.52 seconds: 644.50 ops/second PASS 37 (20s) == sanityn test 39a: file mtime does not change after rename ========================================================== 20:41:31 (1768009291) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 20:41:33 (1768009293) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 20:41:37 (1768009297) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 20:41:40 (1768009300) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 20:41:43 (1768009303) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 20:41:51 (1768009311) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 20:41:59 (1768009319) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 20:42:07 (1768009327) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 20:42:15 (1768009335) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 20:42:20 (1768009340) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 20:42:25 (1768009345) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 20:42:30 (1768009350) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 20:42:35 (1768009355) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 20:42:40 (1768009360) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 20:42:45 (1768009365) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 20:42:50 (1768009370) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 20:42:55 (1768009375) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (721s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 20:54:56 (1768010096) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 20:55:01 (1768010101) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 20:55:06 (1768010106) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 20:55:12 (1768010112) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 20:55:17 (1768010117) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 20:55:23 (1768010123) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 20:55:28 (1768010128) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 20:55:34 (1768010134) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 20:55:39 (1768010139) PASS 43a (20s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 20:55:59 (1768010159) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 20:56:04 (1768010164) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 20:56:09 (1768010169) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 20:56:14 (1768010174) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 20:56:20 (1768010180) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 20:56:25 (1768010185) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 20:56:30 (1768010190) Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 20:56:36 (1768010196) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 20:56:37 (1768010197) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (40s) == sanityn test 43k: unlink vs create ==================== 20:57:17 (1768010237) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (417s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 21:04:14 (1768010654) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Jan 9 21:04:18 EST 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 21:04:19 (1768010659) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 21:04:24 (1768010664) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 21:04:29 (1768010669) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 21:04:34 (1768010674) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 21:04:39 (1768010679) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 21:04:44 (1768010684) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 21:04:49 (1768010689) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 21:04:54 (1768010694) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 21:04:55 (1768010695) PASS 45a (28s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 21:05:23 (1768010723) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 21:05:28 (1768010728) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 21:05:33 (1768010733) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 21:05:38 (1768010738) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 21:05:43 (1768010743) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 21:05:49 (1768010749) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 21:05:54 (1768010754) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 21:05:59 (1768010759) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 21:06:00 (1768010760) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (416s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 21:12:56 (1768011176) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 21:13:01 (1768011181) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 21:13:06 (1768011186) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 21:13:12 (1768011192) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 21:13:17 (1768011197) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 21:13:22 (1768011202) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 21:13:28 (1768011208) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 21:13:33 (1768011213) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 21:13:38 (1768011218) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 21:13:39 (1768011219) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 21:13:40 (1768011220) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 21:13:41 (1768011221) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 21:13:42 (1768011222) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 21:13:43 (1768011223) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 21:13:45 (1768011225) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 21:13:46 (1768011226) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 21:13:47 (1768011227) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00561251 s, 1.8 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:13:54 (1768011234) 0+1 records in 0+1 records out 158 bytes copied, 0.00374298 s, 42.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216415) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 21:13:58 (1768011238) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00345837 s, 296 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188736 s, 55.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:14:13 (1768011253) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00433594 s, 236 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:14:19 (1768011259) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142899 s, 73.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 21:14:23 (1768011263) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 21:14:27 (1768011267) llite.lustre-ffff9aa147d3e800.xattr_cache=1 llite.lustre-ffff9aa14a5b9800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:27.000000000 -0500 Modify: 2026-01-09 21:14:27.000000000 -0500 Change: 2026-01-09 21:14:27.000000000 -0500 Birth: 2026-01-09 21:14:28.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:33.000000000 -0500 Modify: 2026-01-09 21:14:33.000000000 -0500 Change: 2026-01-09 21:14:33.000000000 -0500 Birth: 2026-01-09 21:14:33.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:38.000000000 -0500 Modify: 2026-01-09 21:14:38.000000000 -0500 Change: 2026-01-09 21:14:38.000000000 -0500 Birth: 2026-01-09 21:14:39.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:38.000000000 -0500 Modify: 2026-01-09 21:14:39.000000000 -0500 Change: 2026-01-09 21:14:39.000000000 -0500 Birth: 2026-01-09 21:14:39.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:44.000000000 -0500 Modify: 2026-01-09 21:14:44.000000000 -0500 Change: 2026-01-09 21:14:44.000000000 -0500 Birth: 2026-01-09 21:14:44.000000000 -0500 File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:14:44.000000000 -0500 Modify: 2026-01-09 21:14:44.000000000 -0500 Change: 2026-01-09 21:14:44.000000000 -0500 Birth: 2026-01-09 21:14:44.000000000 -0500 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 21:14:51 (1768011291) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 21:14:59 (1768011299) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 21:15:06 (1768011306) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 21:15:19 (1768011319) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 21:15:28 (1768011328) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 21:15:29 (1768011329) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 21:15:42 (1768011342) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 21:15:56 (1768011356) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1768011356 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.54329 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (26s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 21:16:22 (1768011382) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1768011383 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.69922 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (38s) == sanityn test 60: Verify data_version behaviour ======== 21:17:00 (1768011420) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 21:17:02 (1768011422) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 21:17:04 (1768011424) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 21:17:07 (1768011427) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0048691 s, 13.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 21:17:08 (1768011428) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00701459 s, 9.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 21:17:09 (1768011429) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 21:17:11 (1768011431) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00433223 s, 15.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 21:17:12 (1768011432) llite.lustre-ffff9aa147d3e800.xattr_cache=1 llite.lustre-ffff9aa14a5b9800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 21:17:14 (1768011434) llite.lustre-ffff9aa147d3e800.xattr_cache=1 llite.lustre-ffff9aa14a5b9800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 21:17:17 (1768011437) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00321555 s, 318 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00393859 s, 260 kB/s 229185: get lock2 229186: get lock1 229186: done 229186: get lock2 229185: done 229185: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 229186: cannot get lock2: Resource deadlock avoided 229186: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 229185: cannot get lock1: Resource deadlock avoided 229185: done 229185: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 21:17:23 (1768011443) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0254867 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143019 s, 73.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229766 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 21:17:30 (1768011450) opening files: ........ closing files: ........ PASS 76 (16s) == sanityn test 77a: check FIFO NRS policy =============== 21:17:46 (1768011466) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120772 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329006 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311577 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838947 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963555 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888784 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891663 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934965 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102885 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108599 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980326 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105462 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114994 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998123 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109126 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124311 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101207 s, 10.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136879 s, 76.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171697 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147729 s, 71.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712271 s, 147 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129587 s, 80.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227448 s, 46.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717218 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352544 s, 297 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367247 s, 286 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247928 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123477 s, 84.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028409 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235325 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956921 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115577 s, 90.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195636 s, 53.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 21:17:49 (1768011469) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118013 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271491 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367488 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663969 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676618 s, 15.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103293 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101262 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100321 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11346 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120804 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101898 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111063 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935497 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115944 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127032 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127008 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120792 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010257 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106269 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409678 s, 256 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406428 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0074765 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689029 s, 152 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429048 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275584 s, 380 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989975 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221772 s, 47.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208436 s, 50.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365617 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238811 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349849 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283382 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310185 s, 338 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141356 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472143 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555894 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630267 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666911 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468475 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565248 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629203 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686098 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561169 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649743 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881123 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878651 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10612 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107639 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138801 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128505 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153411 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185241 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173982 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135802 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244765 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114072 s, 91.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339923 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184617 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252249 s, 41.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137199 s, 76.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00653287 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936259 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338884 s, 309 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100814 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348703 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458477 s, 229 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 21:17:53 (1768011473) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137412 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270493 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314684 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568887 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663596 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932718 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636644 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084539 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100995 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940841 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935714 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102812 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879639 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885773 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990738 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103513 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977827 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194321 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135779 s, 77.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218815 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156142 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766584 s, 137 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164734 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453587 s, 231 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241048 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163025 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122957 s, 85.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137718 s, 76.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261565 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278871 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139835 s, 75.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251291 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165587 s, 63.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131972 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275824 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608669 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674535 s, 15.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095111 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100829 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093591 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106961 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938874 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098316 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107443 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102508 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119392 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112966 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113941 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113073 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208082 s, 50.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235719 s, 44.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207873 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208016 s, 50.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491053 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174872 s, 60.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237949 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153657 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024388 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018093 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022922 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127547 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100812 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229452 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00961141 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647513 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707646 s, 148 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124004 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216756 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485299 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432737 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585232 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585644 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622903 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612864 s, 17.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699446 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608957 s, 17.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106151 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11338 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989094 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942549 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124905 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125439 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147108 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203713 s, 51.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086675 s, 121 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345553 s, 30.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379617 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398259 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286651 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175807 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018807 s, 55.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195612 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323733 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145921 s, 71.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345296 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514849 s, 20.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352935 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354816 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259419 s, 40.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=3 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (4s) == sanityn test 77d: check TRR nrs policy ================ 21:17:57 (1768011477) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.170444 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336447 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316765 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499018 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743803 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791017 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763405 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889856 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897427 s, 11.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108557 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104539 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127187 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133361 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140586 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118187 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145393 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151695 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134384 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100223 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166035 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202816 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135791 s, 77.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296408 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154529 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357562 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910763 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176675 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282007 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202663 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020039 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212095 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00259784 s, 404 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460294 s, 228 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159841 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031642 s, 33.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241326 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362075 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786784 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698354 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991201 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10177 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082176 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878823 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104988 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819754 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107057 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116394 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113686 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124962 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105895 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159029 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117172 s, 89.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214653 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223304 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162229 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513781 s, 204 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208657 s, 50.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266885 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125061 s, 83.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117546 s, 89.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018265 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198786 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109151 s, 96.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254796 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117595 s, 89.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196312 s, 53.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121579 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344866 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304786 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093444 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100533 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868006 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866347 s, 12.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972478 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10551 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962546 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104657 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107867 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105447 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109784 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949641 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109645 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125841 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165703 s, 63.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154305 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135445 s, 77.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504314 s, 208 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156723 s, 66.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166764 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198588 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216927 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264244 s, 39.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663604 s, 158 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213616 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909508 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261893 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583217 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373732 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311755 s, 336 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 21:18:02 (1768011482) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.46@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171509 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024272 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363466 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667445 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598839 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739591 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460091 s, 22.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879028 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746559 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854743 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863879 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760056 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10253 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974465 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112577 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132237 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225977 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126048 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135209 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243636 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146657 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125288 s, 83.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185787 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340191 s, 308 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444694 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177699 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672012 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345865 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155091 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246732 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031663 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140882 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0020649 s, 508 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118493 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209558 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301899 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096181 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870391 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850754 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112375 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105216 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901903 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104084 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981198 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881028 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104948 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108566 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110365 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121693 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106383 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127325 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1322 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133161 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135158 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122717 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131296 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135125 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133392 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133975 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138773 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130446 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137504 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132575 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131537 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135547 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134098 s, 7.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131058 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214332 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382355 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497568 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735095 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890043 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810753 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786441 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701653 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827086 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862822 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953659 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941264 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828473 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960139 s, 10.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110917 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107032 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa147d3e800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109177 s, 96.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137959 s, 76.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175278 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157486 s, 66.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191809 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195195 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155673 s, 67.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180376 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272719 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176152 s, 59.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189191 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247248 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244965 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206718 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211188 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233925 s, 44.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113536 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416552 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583021 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670456 s, 15.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632387 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074306 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671473 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546782 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726953 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070955 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865446 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057284 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078865 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841082 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905496 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773009 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820336 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163036 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160182 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148511 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145144 s, 72.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018633 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205321 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189249 s, 55.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180424 s, 58.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0071178 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051638 s, 203 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043279 s, 242 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639935 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253628 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318349 s, 329 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288728 s, 363 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239749 s, 437 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 21:18:09 (1768011489) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123483 s, 136 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023475 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302464 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063263 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498544 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816551 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799559 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941122 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784341 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869171 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820384 s, 12.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088641 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109801 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112199 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108102 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103987 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109303 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116534 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135619 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365963 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192913 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201304 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142363 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242124 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135909 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00212453 s, 494 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022597 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00326603 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420414 s, 249 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214595 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119625 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285776 s, 367 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141557 s, 74.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143917 s, 117 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304765 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408164 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426523 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568739 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476928 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549547 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045859 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503452 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826658 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874022 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555242 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765108 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940056 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846414 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788846 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877903 s, 11.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220546 s, 47.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227611 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017844 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272122 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217168 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217568 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275836 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202589 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022713 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192404 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018565 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028248 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018787 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184881 s, 56.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199919 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257087 s, 40.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123559 s, 136 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329831 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406837 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824478 s, 12.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789355 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102369 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832674 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100427 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952414 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121707 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129366 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106507 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112248 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124088 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138955 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126273 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147145 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197898 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158932 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346026 s, 303 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437727 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227718 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197733 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347752 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036127 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00334278 s, 314 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198561 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321148 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226957 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178616 s, 58.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011237 s, 93.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902378 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231506 s, 45.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134311 s, 125 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246184 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342999 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454799 s, 23.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432971 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624569 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664049 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054539 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704078 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056156 s, 18.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa147d3e800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114913 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112261 s, 9.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125502 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134591 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133963 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138408 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1396 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102984 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134314 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142329 s, 73.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150224 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151394 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149746 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176859 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169053 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172049 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214554 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017576 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225342 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235843 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220646 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213233 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407385 s, 25.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (6s) == sanityn test 77g: Change TBF type directly ============ 21:18:15 (1768011495) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 21:18:19 (1768011499) oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg346-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg346-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg346-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg346-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 21:18:22 (1768011502) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 21:18:29 (1768011509) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.112739 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448387 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061908 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664129 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662964 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847579 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680124 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778123 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741508 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880288 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08085 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806281 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758047 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907694 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771573 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073488 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643384 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007495 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289667 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176127 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144664 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198453 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319885 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264678 s, 39.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632888 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196447 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209009 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433859 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266865 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454697 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278066 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454932 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252998 s, 41.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96241 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4317 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (49s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 21:19:18 (1768011558) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455306 s, 36.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485372 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647401 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667309 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395153 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605184 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696312 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504736 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598545 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108317 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120442 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131183 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164744 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16875 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15707 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164303 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019863 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208948 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153773 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213834 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018859 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451665 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117238 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178151 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269668 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916891 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708324 s, 148 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958908 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010157 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687613 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457353 s, 229 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.829 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9514 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.458334 s, 36.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441244 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476303 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361482 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591067 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512728 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470219 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626985 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229366 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119171 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120638 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134492 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131318 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159624 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145916 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146593 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171727 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184338 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131583 s, 79.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240826 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016832 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176013 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267153 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135947 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215099 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180358 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205179 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190123 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246487 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631063 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427405 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889748 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044223 s, 237 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8288 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9594 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:21:31 (1768011691) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22394 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739453 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573443 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074328 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817933 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638112 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101436 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101173 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110583 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160405 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160392 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162964 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158263 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163982 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149583 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154549 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188884 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146552 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151576 s, 69.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204197 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221917 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255676 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264638 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273582 s, 38.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265564 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217518 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01722 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227065 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622002 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220594 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026662 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267874 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373012 s, 281 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8345 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9469 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22671 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491044 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902069 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702709 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819078 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825563 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091496 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889436 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966812 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152469 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152442 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131386 s, 8.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171053 s, 6.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166485 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163534 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160258 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16081 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120554 s, 87.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184745 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020001 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210689 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156959 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205696 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212297 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177547 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354424 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00778382 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309511 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258478 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255326 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424756 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547342 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506988 s, 207 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8317 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9537 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (132s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:23:43 (1768011823) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247723 s, 67.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345529 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686286 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816741 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820709 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864205 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773051 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782316 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837113 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073694 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934025 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855455 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831095 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935696 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750547 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862648 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106877 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169084 s, 62.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147903 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196891 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183595 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168707 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134173 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185136 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422865 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171918 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215623 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221078 s, 47.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234944 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233313 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843404 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293822 s, 357 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195388 s, 53.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96139 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72987 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.46@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.46@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122606 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209449 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04355 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655908 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658858 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669027 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495972 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591098 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581039 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673907 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083859 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160004 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166876 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219347 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196745 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242583 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258067 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131165 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187137 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168454 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142817 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181662 s, 57.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176955 s, 59.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183617 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199543 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196905 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443711 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105675 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011057 s, 94.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431957 s, 243 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498358 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009939 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00964668 s, 109 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96097 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73169 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.46@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127264 s, 132 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273805 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466736 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755492 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565958 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808034 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962065 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766082 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987038 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889491 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896488 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886249 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851881 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103486 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900723 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949561 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109692 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285572 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235964 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036236 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02547 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216508 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131401 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384001 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332535 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228171 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405635 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358147 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449432 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339028 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396374 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507616 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581665 s, 180 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94479 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88149 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131825 s, 127 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333471 s, 31.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263557 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103179 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873672 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109428 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953321 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101797 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992215 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973453 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112242 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114445 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101895 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130714 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145247 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206856 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217862 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147903 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168308 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138413 s, 75.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147811 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151807 s, 69.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211274 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452344 s, 232 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214389 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458155 s, 229 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00773382 s, 136 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450273 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390031 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223194 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109306 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162701 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00200382 s, 523 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77774 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72888 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9076 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73268 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130529 s, 129 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307913 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405159 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311302 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613029 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776187 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721461 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737715 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613931 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827869 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872842 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990872 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10648 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114474 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121164 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11697 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130453 s, 8.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139577 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192685 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164098 s, 63.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018055 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181426 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012867 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416253 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403531 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211013 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167105 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224487 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588021 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146677 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237855 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323128 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00345613 s, 303 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6626 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4309 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144032 s, 116 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308517 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602335 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838536 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891881 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953652 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863883 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977353 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830517 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694513 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813175 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874111 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924635 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984959 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973059 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100113 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951952 s, 11.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463063 s, 226 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161989 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151832 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163134 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166741 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183058 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238438 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196767 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190508 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00509398 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232585 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224588 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226299 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468171 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321192 s, 326 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305228 s, 344 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96922 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73138 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13929 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281761 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297486 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102376 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103927 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896203 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835329 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926108 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107368 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101678 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999243 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984647 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102085 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174272 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181606 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205802 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227449 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016257 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201798 s, 52.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158736 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160181 s, 65.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207753 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275674 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183046 s, 57.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128546 s, 81.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260416 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362804 s, 289 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237493 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111246 s, 94.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250156 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868413 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318527 s, 329 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822089 s, 128 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77811 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73194 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76068 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73037 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (355s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:29:38 (1768012178) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130591 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743812 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763857 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816668 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731373 s, 14.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776381 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862652 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980727 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106677 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970864 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082413 s, 12.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965654 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090192 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904231 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871943 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967077 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110765 s, 9.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104089 s, 101 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147298 s, 71.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186105 s, 56.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165768 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162206 s, 64.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424718 s, 247 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216984 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241675 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953467 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198354 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187625 s, 55.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217353 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228356 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430178 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419047 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340377 s, 308 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92712 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73278 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:30:14 (1768012214) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.46@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.46@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117898 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248133 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303954 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103362 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082728 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874138 s, 12.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969001 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947104 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102567 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108434 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106289 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100355 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121564 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187969 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180183 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216656 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23955 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113767 s, 92.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135073 s, 77.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134298 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203393 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123565 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205168 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119004 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237958 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169063 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226009 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594827 s, 176 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118212 s, 88.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639687 s, 164 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110727 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079927 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00192026 s, 546 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9204 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72783 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (36s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:30:50 (1768012250) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.46@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.117056 s, 143 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233086 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743253 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808028 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747999 s, 14.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627027 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075261 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752638 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835412 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887883 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566513 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869617 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713226 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111736 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102078 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999926 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947948 s, 11.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165674 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219757 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246139 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207925 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252768 s, 41.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173106 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249124 s, 42.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113516 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216703 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192463 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205889 s, 50.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229945 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020273 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293185 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247854 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266332 s, 39.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93909 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.968 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (31s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:31:21 (1768012281) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121255 s, 138 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199304 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448315 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709951 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650257 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988493 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630778 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931945 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933347 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101169 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759614 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946993 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949877 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917052 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991615 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190064 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180297 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171886 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144447 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160887 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167227 s, 62.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217722 s, 48.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174438 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113678 s, 92.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216861 s, 48.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211196 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462043 s, 227 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151211 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407906 s, 257 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489716 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00892155 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528193 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00153801 s, 682 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78059 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72921 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91203 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73239 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:32:38 (1768012358) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.850629 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279005 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612119 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556343 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053497 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04745 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055309 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526834 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556519 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161876 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192937 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191333 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185099 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190159 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198805 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189131 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208872 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152313 s, 68.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180806 s, 58.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149538 s, 70.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187945 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202517 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222356 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212347 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00582053 s, 180 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254325 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215809 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129068 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813325 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113703 s, 92.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089142 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259021 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821439 s, 128 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6682 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9556 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:33:45 (1768012425) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251555 s, 66.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316385 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029058 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883093 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843082 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965018 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990813 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119187 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991413 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113575 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108201 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115646 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985841 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108435 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114194 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973902 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121894 s, 8.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110884 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178777 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185355 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912625 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146958 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186755 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160979 s, 65.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261448 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020415 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408525 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474256 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235813 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389331 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00454865 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689219 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00189212 s, 554 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98588 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73168 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg346-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1313 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270792 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362406 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499575 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505411 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523951 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858842 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613696 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881819 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106905 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859598 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980815 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976004 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945031 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088517 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226379 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223095 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145715 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193492 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180475 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164684 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410821 s, 255 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430835 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022929 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620584 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207664 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227585 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265211 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244104 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223586 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278422 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022779 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712567 s, 147 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78284 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73178 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76075 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73105 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (117s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:35:42 (1768012542) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre opts: Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg346-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg346-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.46,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg346-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.46,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg346-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.46,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/59100/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.46,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg346-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Started clients oleg346-client.virtnet: 192.168.203.146@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Started clients oleg346-client.virtnet: 192.168.203.146@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9aa144a87000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa179012000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa144a87000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa179012000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241953 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287442 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663681 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666096 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758664 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594697 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803261 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892069 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958092 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135859 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124911 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259397 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253157 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291486 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301472 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334228 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357337 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136294 s, 76.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205284 s, 51.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220856 s, 47.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167441 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269416 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159087 s, 65.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226816 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017518 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246733 s, 42.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027125 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217823 s, 48.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171993 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263676 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222922 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004476 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249114 s, 42.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.968 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96854 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249881 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027516 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390392 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777084 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992498 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875727 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861751 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777649 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111678 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10681 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929783 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948089 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997075 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264971 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255483 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298066 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299941 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188893 s, 55.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209725 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01808 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194063 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184786 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438151 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127813 s, 82.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220014 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211277 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260929 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418167 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239981 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256387 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267481 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572872 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504008 s, 208 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg346-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg346-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98148 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73294 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (117s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:37:39 (1768012659) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg346-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:37:47 (1768012667) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:37:51 (1768012671) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg346-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.52712 s, 297 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg346-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0134 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:38:42 (1768012722) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130909 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249912 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560276 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580149 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612418 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587297 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729101 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710545 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698255 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433658 s, 24.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa179012000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110877 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103732 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100867 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109052 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104878 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104559 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117406 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219063 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198029 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223269 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409437 s, 256 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106178 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023439 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256847 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243435 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142381 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240468 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255771 s, 41.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222406 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118087 s, 88.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808508 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507374 s, 207 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa179012000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040671 s, 258 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93958 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97459 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14928 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433351 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410005 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154927 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158803 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166184 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152905 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154106 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166256 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166708 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169201 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172603 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164841 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158286 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193957 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189426 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193149 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872831 s, 120 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259518 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177132 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250371 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028908 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031375 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317959 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299741 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333474 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111933 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413524 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347918 s, 30.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315104 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164134 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412374 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387743 s, 27.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg346-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg346-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02867 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96873 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (69s) == sanityn test 77o: Changing rank should not panic ====== 21:39:52 (1768012792) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:39:55 (1768012795) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (40s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:40:35 (1768012835) mds.MDS.mdt.nrs_policies=tbf oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg346-client: oleg346-server: ssh exited with exit code 36 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg346-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:40:47 (1768012847) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 270315 kill md_thread xargs PASS 77r (41s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 21:41:28 (1768012888) oleg346-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg346-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 21:41:31 (1768012891) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935212 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:41:31.000000000 -0500 Modify: 2026-01-09 21:41:31.000000000 -0500 Change: 2026-01-09 21:41:31.000000000 -0500 Birth: 2026-01-09 21:41:32.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 21:41:43 (1768012903) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 21:41:44 (1768012904) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 21:41:45 (1768012905) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 21:41:46 (1768012906) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 21:41:47 (1768012907) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 21:41:48 (1768012908) total: 2500 open/close in 4.96 seconds: 504.06 ops/second snapshot_time 1768012931.542199790 secs.nsecs start_time 1768012910.121725299 secs.nsecs elapsed_time 21.420474491 secs.nsecs open 2505 samples [usecs] 100 2459 732550 235729958 close 2504 samples [usecs] 64 241 238255 24035787 mknod 2501 samples [usecs] 164 2256 636377 180557405 unlink 2500 samples [usecs] 273 9986 1072321 765722865 mkdir 2 samples [usecs] 681 829 1510 1151002 rmdir 2 samples [usecs] 1691 2830 4521 10868381 rename 2500 samples [usecs] 248 4250 2890770 4703504198 getattr 3760 samples [usecs] 7 2916 82487 17246893 parallel_rename_file 2500 samples [usecs] 248 4250 2890770 4703504198 crossdir_rename 2500 samples [usecs] 248 4250 2890770 4703504198 PASS 81d (24s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 21:42:12 (1768012932) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7536 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7536 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 21:42:15 (1768012935) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 21:42:16 (1768012936) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 21:42:23 (1768012943) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1768012944 Finishing test test1 at 1768012944 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 21:42:26 (1768012946) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 21:42:27 (1768012947) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 21:42:28 (1768012948) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 21:42:30 (1768012950) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5880 0x16f8 0x240000400 1 5844 0x16d4 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5881 0x16f9 0x240000400 1 5845 0x16d5 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 21:42:39 (1768012959) osc.lustre-OST0000-osc-ffff9aa144a87000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9aa179012000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9aa144a87000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9aa179012000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0333351 s, 246 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08891 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 288594 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0135882 s, 301 kB/s osc.lustre-OST0000-osc-ffff9aa144a87000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa179012000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa144a87000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa179012000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 21:42:49 (1768012969) llite.lustre-ffff9aa144a87000.fast_read=0 llite.lustre-ffff9aa179012000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0188916 s, 111 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9aa144a87000.fast_read=1 llite.lustre-ffff9aa179012000.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 21:43:04 (1768012984) llite.lustre-ffff9aa144a87000.fast_read=0 llite.lustre-ffff9aa179012000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00211831 s, 5.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9aa144a87000.fast_read=1 llite.lustre-ffff9aa179012000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 21:43:12 (1768012992) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (0s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 21:43:12 (1768012992) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00133691 s, 3.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 21:43:15 (1768012995) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0601927 s, 34.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 21:43:17 (1768012997) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0439447 s, 47.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 21:43:19 (1768012999) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00167108 s, 7.2 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 21:43:21 (1768013001) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0013923 s, 2.9 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 21:43:23 (1768013003) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00157143 s, 2.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 21:43:25 (1768013005) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00157889 s, 2.6 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 21:43:29 (1768013009) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1768013010 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1768013012 Finishing test test23 at 1768013013 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 21:43:36 (1768013016) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 21:43:37 (1768013017) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 21:43:55 (1768013035) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 21:43:56 (1768013036) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00298346 s, 343 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937747 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:43:56.000000000 -0500 Modify: 2026-01-09 21:43:56.000000000 -0500 Change: 2026-01-09 21:43:56.000000000 -0500 Birth: 2026-01-09 21:43:57.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937747 -rw-r--r-- 1 0 0 1768013037 1768013036 1768013036 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937747 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:43:56.000000000 -0500 Modify: 2026-01-09 21:43:56.000000000 -0500 Change: 2026-01-09 21:43:56.000000000 -0500 Birth: 2026-01-09 21:43:57.000000000 -0500 1768013036 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 21:43:58 (1768013038) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 21:44:01 (1768013041) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.109328 s, 95.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5851 0x16db 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:44:06 (1768013046) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0964602 s, 109 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5890 0x1702 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7536 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 21:44:15 (1768013055) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00150469 s, 6.6 MB/s llite.lustre-ffff9aa144a87000.hybrid_io=0 llite.lustre-ffff9aa179012000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.27579 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 21:44:21 (1768013061) 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 192.168.203.146@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg346-client.virtnet /mnt/lustre (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre2 PASS 109 (483s) == sanityn test 110: do not grant another lock on resend ========================================================== 21:52:24 (1768013544) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 21:52:24.000000000 -0500 Modify: 2026-01-09 21:52:24.000000000 -0500 Change: 2026-01-09 21:52:24.000000000 -0500 Birth: 2026-01-09 21:52:25.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg346-client.virtnet oleg346-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 21:53:29 (1768013609) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 21:53:30 (1768013610) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 21:53:31 (1768013611) lustre-OST0000-osc-ffff9aa1462ab000 active. lustre-OST0001-osc-ffff9aa1462ab000 active. lustre-OST0000-osc-ffff9aa1462ab000 active. lustre-OST0001-osc-ffff9aa1462ab000 active. MGC192.168.203.146@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 21:53:33 (1768013613) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 21:53:34 (1768013614) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 21:53:35 (1768013615) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 21:53:36 (1768013616) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00330778 s, 6.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.012371 s, 1.7 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 21:53:40 (1768013620) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=7 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0878 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0861 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0876 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0924 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0903 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1523 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1488 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1614 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1582 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1546 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2373 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2437 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2546 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2527 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa1462ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa1462ab000.ost_server_uuid in FULL state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 5446 sec ============== 21:54:47 (1768013687) === sanityn: start cleanup 21:54:47 (1768013687) === Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg346-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 21:55:27 (1768013727) ===