-----============= acceptance-small: sanityn ============----- Fri Aug 21 22:00:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg225-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 22:01:02 (1787364062) === Starting client oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 Mount client oleg225-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 Started clients oleg225-client.virtnet: 192.168.202.125@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg225-client.virtnet: executing check_config_client /mnt/lustre oleg225-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg225-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a0346458000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9a034cb68000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a0346458000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a034cb68000.idle_timeout=debug disable quota as required oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 22:01:44 (1787364104) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424165 s, 24.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 22:01:47 (1787364107) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 22:01:57 (1787364117) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 22:02:05 (1787364125) -rw-r--r-- 1 root root 0 Aug 21 22:02 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (11s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 22:02:16 (1787364136) -rw-r--r-- 1 root root 0 Aug 21 22:02 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (10s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 22:02:27 (1787364147) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 22:02:34 (1787364154) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 22:02:41 (1787364161) total 1 -rwxrwxrwx 1 root root 0 Aug 21 22:02 f2a -rwxrwxrwx 1 root root 0 Aug 21 22:02 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 21 22:02 f2a -rwxrwxrwx 1 root root 0 Aug 21 22:02 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.012012 s, 42.6 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 22:02:49 (1787364169) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 22:02:53 (1787364173) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.345903 s, 6.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 22:03:00 (1787364180) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 22:03:06 (1787364186) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 22:03:14 (1787364194) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 22:03:22 (1787364202) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 22:03:30 (1787364210) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 22:03:37 (1787364217) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 22:03:46 (1787364226) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 22:03:54 (1787364234) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448964 s, 23.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0644512 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 22:04:01 (1787364241) 1+0 records in 1+0 records out 1 byte copied, 0.00907534 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0223125 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118825 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184132 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011271 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00650114 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0316645 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121583 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.012329 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00971071 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011435 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0094688 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 22:04:08 (1787364248) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0122352 s, 251 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0430203 s, 95.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000637419 s, 4.8 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 22:04:16 (1787364256) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 22:04:23 (1787364263) warning: '-runtime' deprecated, use '-t runtime' instead - link 305 (time 1787364275.95 total 10.03 last 30.42) - link 572 (time 1787364285.99 total 20.07 last 26.58) - link 884 (time 1787364295.99 total 30.07 last 31.19) - link 1194 (time 1787364306.02 total 40.10 last 30.91) - link 1505 (time 1787364316.02 total 50.11 last 31.09) - link 1790 (time 1787364326.05 total 60.13 last 28.43) - link 2094 (time 1787364336.05 total 70.13 last 30.39) - link 2385 (time 1787364346.09 total 80.17 last 29.00) - link 2693 (time 1787364356.10 total 90.18 last 30.77) total: 3015 link in 100.03 seconds: 30.14 ops/second using seed 2666991336 running for 100 seconds - stat 4484 (time 1787364278 ; total 11 ; last 11) - stat 9490 (time 1787364289 ; total 22 ; last 11) - stat 10000 (time 1787364290 ; total 23 ; last 1) - stat 15630 (time 1787364301 ; total 34 ; last 11) - stat 20000 (time 1787364310 ; total 43 ; last 9) - stat 24428 (time 1787364321 ; total 54 ; last 11) - stat 26706 (time 1787364332 ; total 65 ; last 11) - stat 29623 (time 1787364343 ; total 76 ; last 11) - stat 30000 (time 1787364345 ; total 78 ; last 2) - stat 32487 (time 1787364356 ; total 89 ; last 11) - stat 35528 (time 1787364367 ; total 100 ; last 11) total: 35529 stats in 100 seconds: 355.290009 stats/second - unlinked 0 (time 1787364316 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3015) error: No such file or directory total: 3014 unlinks in 85 seconds: 35.458824 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21452) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21454) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 22:06:48 (1787364408) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 22:06:57 (1787364417) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 22639 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 22:07:05 (1787364425) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 23220 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 22:07:13 (1787364433) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23800 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 22:07:21 (1787364441) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24382 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 22:07:30 (1787364450) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24965 Terminated $DIR1/$tdir/sleep 60 PASS 14d (14s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 22:07:44 (1787364464) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (59s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 22:08:44 (1787364524) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.541883 s, 15.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.423748 s, 19.8 MB/s Chance of close/open is 1 in 50 Seed set to 4528 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[1] 1787364529.825562 mapwrite 0x54f5ca thru 0x558879 (0x92b0 bytes) 000004[1] 1787364530.074953 write 0x0768ef thru 0x078bf4 (0x2306 bytes) 000006[1] 1787364530.146328 read 0x3eef77 thru 0x3fa4d8 (0xb562 bytes) 000007[0] 1787364530.155043 mapread 0x2849fa thru 0x293fc7 (0xf5ce bytes) 000009[0] 1787364530.260917 read 0x2adeed thru 0x2b3816 (0x592a bytes) 000010[1] 1787364530.279284 mapwrite 0x80810c thru 0x810140 (0x8035 bytes) 000011[1] 1787364530.464438 mapread 0x284a57 thru 0x291b41 (0xd0eb bytes) 000012[0] 1787364530.535628 mapwrite 0x8a43d4 thru 0x8a699a (0x25c7 bytes) 000015[1] 1787364530.718750 mapread 0x0b5656 thru 0x0bf5f2 (0x9f9d bytes) 000018[0] 1787364530.782152 write 0x26faf8 thru 0x27e9e3 (0xeeec bytes) 000019[0] 1787364530.836327 read 0x237b10 thru 0x2439e9 (0xbeda bytes) 000021[0] 1787364530.847107 read 0x60ed5f thru 0x614790 (0x5a32 bytes) 000024[1] 1787364530.903488 write 0x1d221d thru 0x1d8f31 (0x6d15 bytes) 000025[1] 1787364530.955673 read 0x69adaf thru 0x6a0958 (0x5baa bytes) 000029[0] 1787364531.008053 mapread 0x53b0ee thru 0x5414db (0x63ee bytes) truncating to largest ever: 0x354ba9 000030[1] 1787364531.037905 trunc from 0x8a699b to 0x354ba8 (0x551df2 bytes) 000031[0] 1787364531.096651 trunc from 0x354ba9 to 0x2f1cd8 (0x62ed0 bytes) truncating to largest ever: 0x64abf0 000032[0] 1787364531.153436 trunc from 0x2f1cd9 to 0x64abef (0x358f17 bytes) 000033[0] 1787364531.169851 read 0x1c4056 thru 0x1c46cd (0x678 bytes) 000034[0] 1787364531.176346 mapwrite 0x94e48a thru 0x95d032 (0xeba9 bytes) 000040[0] 1787364531.464599 mapread 0x53f032 thru 0x5490a3 (0xa072 bytes) 000041[1] 1787364531.585630 mapwrite 0x11e375 thru 0x12b940 (0xd5cc bytes) 000043[0] 1787364531.752925 read 0x172844 thru 0x17d6a9 (0xae66 bytes) 000045[0] 1787364531.823748 write 0x5127a8 thru 0x512e1a (0x673 bytes) 000046[1] 1787364531.851767 write 0x1450bd thru 0x14edf1 (0x9d35 bytes) 000048[0] 1787364531.942496 write 0x76d447 thru 0x77c566 (0xf120 bytes) 000051[0] 1787364532.087715 mapwrite 0x7c084d thru 0x7cbf5b (0xb70f bytes) 000054[0] 1787364532.499308 mapwrite 0x3717cd thru 0x37aa7e (0x92b2 bytes) 000056[1] 1787364532.752824 write 0x90dc1d thru 0x918418 (0xa7fc bytes) 000058[1] 1787364532.882954 mapread 0x93c800 thru 0x93ed87 (0x2588 bytes) 000061[1] 1787364533.044041 read 0x8741a0 thru 0x87c603 (0x8464 bytes) 000062[0] 1787364533.073868 mapread 0x170406 thru 0x173894 (0x348f bytes) 000063[1] 1787364533.217552 write 0x63100d thru 0x6400fe (0xf0f2 bytes) 000066[0] 1787364533.331335 read 0x02afa8 thru 0x03a4fd (0xf556 bytes) 000067[1] 1787364533.394611 write 0x5742df thru 0x579ac8 (0x57ea bytes) 000075[0] 1787364533.491939 read 0x65f727 thru 0x6633c0 (0x3c9a bytes) 000077[1] 1787364533.543823 read 0x1d8ab2 thru 0x1e39cc (0xaf1b bytes) 000078[1] 1787364533.557165 trunc from 0x95d033 to 0x60cbc4 (0x35046e bytes) 000079[1] 1787364533.676064 mapread 0x2edf55 thru 0x2eea14 (0xac0 bytes) 000081[1] 1787364533.745964 trunc from 0x60cbc5 to 0x0f6901 (0x5162c3 bytes) 000085[1] 1787364533.879157 read 0x061164 thru 0x06b3ec (0xa289 bytes) 000087[0] 1787364533.983038 write 0x824007 thru 0x829f0c (0x5f06 bytes) 000090[1] 1787364534.143400 mapwrite 0x6e5d35 thru 0x6e9354 (0x3620 bytes) 000091[0] 1787364534.650965 mapread 0x1810c0 thru 0x183498 (0x23d9 bytes) 000092[1] 1787364534.749709 read 0x0237b1 thru 0x0298f1 (0x6141 bytes) 000093[1] 1787364534.780985 trunc from 0x829f0d to 0x031f32 (0x7f7fda bytes) 000094[0] 1787364534.863080 mapwrite 0x32f748 thru 0x33abff (0xb4b8 bytes) truncating to largest ever: 0x77cc12 000096[0] 1787364535.305105 trunc from 0x33ac00 to 0x77cc11 (0x442012 bytes) truncating to largest ever: 0x9e55a3 000099[0] 1787364535.394108 trunc from 0x77cc12 to 0x9e55a2 (0x268991 bytes) 000100[1] 1787364535.486865 mapread 0x04a359 thru 0x0566dd (0xc385 bytes) 000102[0] 1787364535.666012 mapwrite 0x4c09e9 thru 0x4c72c5 (0x68dd bytes) 000103[0] 1787364535.881443 trunc from 0x9e55a3 to 0x85229d (0x193305 bytes) 000107[1] 1787364536.023705 mapread 0x5cbdbe thru 0x5d1cc7 (0x5f0a bytes) 000108[0] 1787364536.081762 trunc from 0x85229e to 0x093b40 (0x7be75d bytes) 000109[0] 1787364536.133661 mapwrite 0x80c4ac thru 0x81be41 (0xf996 bytes) 000111[1] 1787364536.549520 mapread 0x053f5e thru 0x05773b (0x37de bytes) 000117[1] 1787364536.715191 trunc from 0x81be42 to 0x19f344 (0x67cafd bytes) 000118[0] 1787364536.797149 trunc from 0x19f345 to 0x36f0a4 (0x1cfd60 bytes) 000123[1] 1787364537.149111 trunc from 0x36f0a5 to 0x57eff3 (0x20ff4f bytes) 000124[0] 1787364537.290784 read 0x02ca02 thru 0x0313fd (0x49fc bytes) 000125[0] 1787364537.362996 trunc from 0x57eff4 to 0x895767 (0x316774 bytes) 000129[0] 1787364537.512567 mapread 0x2de7e9 thru 0x2e27b8 (0x3fd0 bytes) 000130[0] 1787364537.532674 read 0x1c20f0 thru 0x1ca8f2 (0x8803 bytes) 000131[0] 1787364537.566741 read 0x88c954 thru 0x892eda (0x6587 bytes) 000133[0] 1787364537.625911 mapread 0x23532f thru 0x240da8 (0xba7a bytes) 000135[0] 1787364537.689317 write 0x88f82f thru 0x893527 (0x3cf9 bytes) 000138[1] 1787364537.789146 trunc from 0x895768 to 0x28df54 (0x607813 bytes) 000139[0] 1787364537.956764 write 0x100d2d thru 0x10460e (0x38e2 bytes) 000141[1] 1787364538.121315 trunc from 0x28df55 to 0x5b618c (0x328238 bytes) 000142[0] 1787364538.253405 mapwrite 0x1a8d4c thru 0x1b8207 (0xf4bc bytes) 000145[1] 1787364538.503728 read 0x16cc2b thru 0x17be63 (0xf239 bytes) 000146[0] 1787364538.592440 write 0x5a3451 thru 0x5afee0 (0xca90 bytes) 000147[1] 1787364538.669737 write 0x707399 thru 0x7137ba (0xc422 bytes) 000148[0] 1787364538.754661 mapread 0x125e80 thru 0x129c94 (0x3e15 bytes) 000149[0] 1787364538.818816 write 0x959b0d thru 0x966dd6 (0xd2ca bytes) 000152[1] 1787364538.927544 read 0x83e906 thru 0x84d59b (0xec96 bytes) 000153[1] 1787364538.967679 write 0x63a9a7 thru 0x64777d (0xcdd7 bytes) 000154[0] 1787364538.980509 write 0x588bc3 thru 0x594b81 (0xbfbf bytes) 000155[1] 1787364539.059181 write 0x9ca1c7 thru 0x9cec62 (0x4a9c bytes) 000156[0] 1787364539.126398 mapread 0x061462 thru 0x0666c3 (0x5262 bytes) 000157[1] 1787364539.210933 mapwrite 0x317615 thru 0x317d33 (0x71f bytes) 000162[0] 1787364539.459287 trunc from 0x9cec63 to 0x2ae598 (0x7206ca bytes) 000163[1] 1787364539.532186 mapread 0x0720ae thru 0x076eba (0x4e0d bytes) 000164[0] 1787364539.626401 write 0x3ccf6e thru 0x3ce82a (0x18bd bytes) 000168[0] 1787364539.749774 mapread 0x0ddd0f thru 0x0df307 (0x15f9 bytes) 000170[0] 1787364539.825110 write 0x7049aa thru 0x70dfa5 (0x95fc bytes) 000171[0] 1787364539.851569 mapwrite 0x6dba92 thru 0x6dbcb5 (0x224 bytes) 000172[0] 1787364539.948596 read 0x2fa54b thru 0x308bf9 (0xe6af bytes) 000173[0] 1787364539.958972 trunc from 0x70dfa6 to 0x001440 (0x70cb65 bytes) 000174[1] 1787364540.041884 mapwrite 0x0c5870 thru 0x0ceca6 (0x9437 bytes) 000176[0] 1787364540.260697 mapread 0x057124 thru 0x05d2f1 (0x61ce bytes) 000177[0] 1787364540.363355 mapwrite 0x562d9d thru 0x565552 (0x27b6 bytes) 000180[0] 1787364540.569446 trunc from 0x565553 to 0x6e9d2d (0x1847db bytes) 000183[1] 1787364540.705838 trunc from 0x6e9d2e to 0x33cb74 (0x3ad1b9 bytes) 000187[0] 1787364540.805904 mapwrite 0x706f0c thru 0x71546c (0xe561 bytes) 000193[1] 1787364541.020472 trunc from 0x71546d to 0x1040e9 (0x611383 bytes) 000197[0] 1787364541.115668 trunc from 0x1040ea to 0x7722ff (0x66e216 bytes) 000198[0] 1787364541.192292 mapread 0x129f2e thru 0x12f879 (0x594c bytes) 000199[0] 1787364541.242950 mapread 0x367291 thru 0x36f351 (0x80c1 bytes) 000200[1] 1787364541.294961 mapwrite 0x2b9aca thru 0x2c79ca (0xdf01 bytes) 000201[1] 1787364541.400187 mapwrite 0x3a8708 thru 0x3b4cf3 (0xc5ec bytes) 000205[0] 1787364541.527969 mapwrite 0x86ab43 thru 0x86d804 (0x2cc2 bytes) 000207[1] 1787364541.868173 mapread 0x144c8e thru 0x14d6a4 (0x8a17 bytes) 000208[0] 1787364542.025043 mapread 0x6eec27 thru 0x6fcf85 (0xe35f bytes) 000210[0] 1787364542.127793 mapwrite 0x764439 thru 0x76850f (0x40d7 bytes) 000211[1] 1787364542.289641 trunc from 0x86d805 to 0x14eeb6 (0x71e94e bytes) 000213[0] 1787364542.389695 mapwrite 0x1ed835 thru 0x1f37dc (0x5fa8 bytes) 000216[0] 1787364542.689583 trunc from 0x1f37dd to 0x2ada72 (0xba296 bytes) 000218[0] 1787364542.819334 read 0x071f95 thru 0x07f53a (0xd5a6 bytes) 000219[0] 1787364542.961715 mapwrite 0x1513e6 thru 0x15c978 (0xb593 bytes) truncating to largest ever: 0x9ec7c8 000222[1] 1787364543.208806 trunc from 0x2ada73 to 0x9ec7c7 (0x73ed55 bytes) 000223[0] 1787364543.354518 mapread 0x55840c thru 0x564ff8 (0xcbed bytes) 000224[1] 1787364543.487382 read 0x1e4286 thru 0x1e9f4e (0x5cc9 bytes) 000228[1] 1787364543.558311 write 0x4ff737 thru 0x50c68d (0xcf57 bytes) 000230[0] 1787364543.639777 read 0x1846cb thru 0x188ff2 (0x4928 bytes) 000236[0] 1787364543.741862 mapread 0x4046d3 thru 0x40a37b (0x5ca9 bytes) 000239[0] 1787364543.851094 read 0x4433bc thru 0x452251 (0xee96 bytes) 000240[1] 1787364543.874177 read 0x84465d thru 0x848fa0 (0x4944 bytes) 000242[0] 1787364543.888950 mapwrite 0x02bec4 thru 0x03922b (0xd368 bytes) 000243[0] 1787364544.060916 trunc from 0x9ec7c8 to 0x78d5ba (0x25f20d bytes) 000246[1] 1787364544.166501 read 0x4f3ad2 thru 0x4f4880 (0xdaf bytes) 000247[0] 1787364544.240764 read 0x57d619 thru 0x581493 (0x3e7b bytes) 000248[0] 1787364544.300966 write 0x38d608 thru 0x397f46 (0xa93f bytes) 000249[0] 1787364544.321735 mapread 0x42fb7d thru 0x4381f5 (0x8679 bytes) 000250[1] 1787364544.349675 trunc from 0x78d5bb to 0x6143fe (0x1791bc bytes) 000255[0] 1787364544.458747 read 0x0f49fa thru 0x0ffe5a (0xb461 bytes) 000256[0] 1787364544.497679 mapread 0x188fbc thru 0x18d72f (0x4774 bytes) 000257[1] 1787364544.528084 mapread 0x344803 thru 0x346d2f (0x252d bytes) 000261[0] 1787364544.568652 read 0x5544ae thru 0x559639 (0x518c bytes) 000263[0] 1787364544.598565 write 0x4c5aa2 thru 0x4d1551 (0xbab0 bytes) 000264[0] 1787364544.628601 mapwrite 0x619f84 thru 0x626a33 (0xcab0 bytes) 000265[1] 1787364544.950848 write 0x1ea96b thru 0x1f2b80 (0x8216 bytes) 000267[1] 1787364545.055620 mapread 0x334284 thru 0x342bf8 (0xe975 bytes) 000268[1] 1787364545.147817 write 0x66a572 thru 0x67050a (0x5f99 bytes) 000269[0] 1787364545.171908 write 0x148cb4 thru 0x156f3d (0xe28a bytes) 000270[1] 1787364545.267290 trunc from 0x67050b to 0x661a03 (0xeb07 bytes) 000275[0] 1787364545.509026 trunc from 0x661a04 to 0x544abe (0x11cf45 bytes) 000279[1] 1787364545.777986 mapread 0x217e9d thru 0x226596 (0xe6fa bytes) 000280[1] 1787364545.935104 write 0x23990b thru 0x247587 (0xdc7d bytes) 000282[1] 1787364546.007396 trunc from 0x544abf to 0x2c61ea (0x27e8d4 bytes) 000284[0] 1787364546.037600 mapread 0x17e9c6 thru 0x181d09 (0x3344 bytes) 000285[0] 1787364546.097696 mapwrite 0x31d60e thru 0x32c845 (0xf238 bytes) 000287[0] 1787364546.316488 write 0x8ab02d thru 0x8b309e (0x8072 bytes) 000289[0] 1787364546.353890 mapread 0x68d1f0 thru 0x690247 (0x3058 bytes) 000290[1] 1787364546.388653 read 0x65d548 thru 0x665405 (0x7ebe bytes) 000292[1] 1787364546.440989 trunc from 0x8b309f to 0x0c5106 (0x7edf98 bytes) 000294[0] 1787364546.531170 mapread 0x091a4e thru 0x096e5c (0x540f bytes) 000295[1] 1787364546.588132 trunc from 0x0c5107 to 0x263803 (0x19e6fd bytes) 000297[0] 1787364546.675470 read 0x14f8c7 thru 0x151911 (0x204b bytes) 000299[0] 1787364546.743552 write 0x46b3d2 thru 0x46d0e2 (0x1d11 bytes) 000302[0] 1787364546.776189 mapread 0x0ae0a9 thru 0x0ae3ae (0x306 bytes) 000304[1] 1787364546.777443 read 0x1216f5 thru 0x12f69f (0xdfab bytes) 000305[1] 1787364546.787387 mapwrite 0x3c619f thru 0x3d32ef (0xd151 bytes) 000307[1] 1787364547.386933 read 0x3d434b thru 0x3d8d78 (0x4a2e bytes) 000308[1] 1787364547.474685 mapread 0x0c18c8 thru 0x0c3a98 (0x21d1 bytes) 000310[0] 1787364547.588778 mapread 0x0fd4cc thru 0x0fe759 (0x128e bytes) 000312[0] 1787364547.673356 read 0x22b256 thru 0x2346b9 (0x9464 bytes) 000314[0] 1787364547.723241 read 0x3ccb1e thru 0x3ce268 (0x174b bytes) 000315[0] 1787364547.757066 mapread 0x0a3a76 thru 0x0b1493 (0xda1e bytes) 000316[1] 1787364547.841865 read 0x0f9b37 thru 0x0fe627 (0x4af1 bytes) 000317[1] 1787364547.906459 trunc from 0x46d0e3 to 0x27eac3 (0x1ee61f bytes) 000318[0] 1787364548.027702 trunc from 0x27eac4 to 0x3afcf0 (0x13122d bytes) 000320[0] 1787364548.117659 read 0x1e4eb7 thru 0x1ea75e (0x58a8 bytes) 000322[1] 1787364548.263033 read 0x2d580e thru 0x2d6492 (0xc85 bytes) 000323[0] 1787364548.358876 read 0x1800f7 thru 0x181ce5 (0x1bef bytes) 000327[0] 1787364548.468070 mapread 0x1c1cc5 thru 0x1c7be1 (0x5f1d bytes) 000328[0] 1787364548.470060 read 0x0b55db thru 0x0b989a (0x42c0 bytes) 000330[0] 1787364548.491006 write 0x238347 thru 0x241816 (0x94d0 bytes) 000336[1] 1787364548.674280 mapread 0x07417d thru 0x07e2d0 (0xa154 bytes) 000339[1] 1787364548.894541 write 0x2fd88d thru 0x30733f (0x9ab3 bytes) 000341[1] 1787364549.080267 read 0x21a82b thru 0x21fc8f (0x5465 bytes) 000344[0] 1787364549.305464 read 0x164c2b thru 0x172d7c (0xe152 bytes) 000346[1] 1787364549.549009 mapwrite 0x90b3e8 thru 0x91a813 (0xf42c bytes) 000350[1] 1787364550.088520 read 0x6a5dc9 thru 0x6a7066 (0x129e bytes) 000351[1] 1787364550.089972 trunc from 0x91a814 to 0x76d756 (0x1ad0bd bytes) 000352[0] 1787364550.152779 read 0x61e94c thru 0x622ab2 (0x4167 bytes) 000357[1] 1787364550.296827 mapread 0x57dbaf thru 0x58081d (0x2c6f bytes) 000358[1] 1787364550.373687 read 0x30aeca thru 0x317098 (0xc1cf bytes) 000359[0] 1787364550.394104 read 0x629425 thru 0x62f16a (0x5d46 bytes) 000361[1] 1787364550.442900 mapread 0x5bc896 thru 0x5c87cb (0xbf36 bytes) 000362[0] 1787364550.503610 read 0x7309f7 thru 0x73cf71 (0xc57b bytes) 000363[0] 1787364550.557367 write 0x4d706a thru 0x4d73c1 (0x358 bytes) 000364[1] 1787364550.612146 trunc from 0x76d757 to 0x286ac9 (0x4e6c8d bytes) 000366[0] 1787364550.739270 mapread 0x14a51d thru 0x1544aa (0x9f8e bytes) 000369[1] 1787364550.902678 read 0x08eb5b thru 0x09bbbb (0xd061 bytes) 000372[1] 1787364551.008498 mapread 0x07afe0 thru 0x086e71 (0xbe92 bytes) 000374[0] 1787364551.023134 mapwrite 0x79f1c1 thru 0x7a88d2 (0x9712 bytes) 000376[1] 1787364551.275969 mapwrite 0x8382cd thru 0x8480e8 (0xfe1c bytes) 000377[0] 1787364551.635965 read 0x07bc5b thru 0x08633f (0xa6e5 bytes) 000378[0] 1787364551.685959 trunc from 0x8480e9 to 0x5a06c0 (0x2a7a28 bytes) 000380[1] 1787364551.865696 trunc from 0x5a06c1 to 0x7f9213 (0x258b53 bytes) 000382[0] 1787364552.000550 mapwrite 0x317582 thru 0x31bde0 (0x485f bytes) 000385[0] 1787364552.266650 trunc from 0x7f9214 to 0x984828 (0x18b615 bytes) 000387[1] 1787364552.334574 trunc from 0x984829 to 0x4cf873 (0x4b4fb5 bytes) 000388[0] 1787364552.432823 mapread 0x4c1fdf thru 0x4c9138 (0x715a bytes) 000392[1] 1787364552.626045 write 0x3e6cb3 thru 0x3f3d51 (0xd09f bytes) 000393[0] 1787364552.646181 mapread 0x17b043 thru 0x17c133 (0x10f1 bytes) 000394[1] 1787364552.715912 mapread 0x086921 thru 0x08df57 (0x7637 bytes) 000395[1] 1787364552.783102 read 0x2bd844 thru 0x2be0c7 (0x884 bytes) 000396[1] 1787364552.792906 read 0x352c41 thru 0x355c5d (0x301d bytes) 000397[1] 1787364552.800604 mapwrite 0x2b2b80 thru 0x2b54e8 (0x2969 bytes) 000398[0] 1787364552.978865 mapread 0x0f4621 thru 0x0f7ffe (0x39de bytes) 000402[0] 1787364553.020502 write 0x619481 thru 0x61f9d9 (0x6559 bytes) 000407[1] 1787364553.091947 write 0x9d1d3a thru 0x9dc027 (0xa2ee bytes) 000408[1] 1787364553.160329 trunc from 0x9dc028 to 0x0e5a60 (0x8f65c7 bytes) 000412[1] 1787364553.374691 mapread 0x0a873f thru 0x0aa773 (0x2035 bytes) 000413[1] 1787364553.473192 trunc from 0x0e5a61 to 0x589502 (0x4a3aa2 bytes) 000415[0] 1787364553.560643 trunc from 0x589503 to 0x7e231f (0x258e1d bytes) 000417[0] 1787364553.754890 write 0x60c2a8 thru 0x61c034 (0xfd8d bytes) 000419[0] 1787364553.781016 read 0x35023c thru 0x353bca (0x398f bytes) 000421[1] 1787364553.827102 trunc from 0x7e2320 to 0x0e8560 (0x6f9dbf bytes) 000422[0] 1787364553.974510 mapwrite 0x2792da thru 0x285115 (0xbe3c bytes) 000423[0] 1787364554.223066 write 0x9f2510 thru 0x9fc338 (0x9e29 bytes) 000426[0] 1787364554.364534 mapwrite 0x6c5d54 thru 0x6cf224 (0x94d1 bytes) 000427[0] 1787364554.589432 mapread 0x595d44 thru 0x59d899 (0x7b56 bytes) 000428[1] 1787364554.629955 mapwrite 0x42d6d0 thru 0x436a96 (0x93c7 bytes) 000430[0] 1787364554.870608 mapwrite 0x1a9c9c thru 0x1b7893 (0xdbf8 bytes) 000432[0] 1787364555.141669 write 0x78761b thru 0x78de34 (0x681a bytes) 000433[1] 1787364555.195634 read 0x7414b4 thru 0x744808 (0x3355 bytes) 000434[1] 1787364555.257337 trunc from 0x9fc339 to 0x7c30a5 (0x239293 bytes) 000435[1] 1787364555.401918 trunc from 0x7c30a6 to 0x30fd4f (0x4b3356 bytes) 000437[0] 1787364555.461767 trunc from 0x30fd50 to 0x1aca73 (0x1632dc bytes) 000438[0] 1787364555.510424 write 0x459d63 thru 0x45caa6 (0x2d44 bytes) 000439[1] 1787364555.512981 read 0x1aeb10 thru 0x1b0365 (0x1856 bytes) 000440[0] 1787364555.608067 write 0x427b92 thru 0x4344ad (0xc91c bytes) 000442[1] 1787364555.621921 trunc from 0x45caa7 to 0x095c93 (0x3c6e13 bytes) 000445[1] 1787364555.751444 trunc from 0x095c94 to 0x5bf2f4 (0x529661 bytes) 000446[0] 1787364555.780377 trunc from 0x5bf2f5 to 0x7d413a (0x214e46 bytes) 000447[0] 1787364555.843062 write 0x39f05f thru 0x3a61a9 (0x714b bytes) 000448[0] 1787364555.874343 mapwrite 0x48eed0 thru 0x48f4c8 (0x5f9 bytes) 000455[0] 1787364556.061623 write 0x0dbba3 thru 0x0e191a (0x5d78 bytes) 000458[0] 1787364556.195760 trunc from 0x7d413b to 0x3d212d (0x40200d bytes) 000459[1] 1787364556.254842 mapwrite 0x5c9be2 thru 0x5ccd61 (0x3180 bytes) 000460[1] 1787364556.520383 write 0x319a34 thru 0x327074 (0xd641 bytes) 000461[0] 1787364556.523179 mapread 0x047d19 thru 0x05365f (0xb947 bytes) 000462[1] 1787364556.617033 trunc from 0x5ccd62 to 0x8c8058 (0x2fb2f7 bytes) 000466[1] 1787364556.702280 read 0x42eb14 thru 0x431394 (0x2881 bytes) 000471[1] 1787364556.826847 trunc from 0x8c8059 to 0x017a09 (0x8b064f bytes) truncating to largest ever: 0x9fbe03 000477[0] 1787364556.941410 trunc from 0x017a0a to 0x9fbe02 (0x9e43f9 bytes) 000479[1] 1787364557.023561 read 0x5ed179 thru 0x5fbe9d (0xed25 bytes) 000480[1] 1787364557.071310 trunc from 0x9fbe03 to 0x05178c (0x9aa676 bytes) 000482[1] 1787364557.161908 write 0x9b1ae6 thru 0x9b1e2e (0x349 bytes) 000483[0] 1787364557.188313 mapread 0x8de39f thru 0x8eca6f (0xe6d1 bytes) 000487[1] 1787364557.284996 trunc from 0x9b1e2f to 0x6b8a57 (0x2f93d7 bytes) 000488[1] 1787364557.343190 trunc from 0x6b8a58 to 0x198e12 (0x51fc45 bytes) 000489[0] 1787364557.403903 trunc from 0x198e13 to 0x4247f3 (0x28b9e1 bytes) 000490[0] 1787364557.467879 trunc from 0x4247f4 to 0x5df18c (0x1ba999 bytes) 000492[0] 1787364557.492136 write 0x82678e thru 0x8284df (0x1d52 bytes) 000493[1] 1787364557.525870 read 0x70cb9b thru 0x710824 (0x3c8a bytes) 000494[0] 1787364557.598117 mapwrite 0x329c56 thru 0x32d7cb (0x3b76 bytes) 000495[0] 1787364557.731025 mapread 0x29a177 thru 0x2a3002 (0x8e8c bytes) 000496[1] 1787364557.759734 read 0x5c1b47 thru 0x5c7b5d (0x6017 bytes) 000497[1] 1787364557.777754 write 0x249bc1 thru 0x24fafd (0x5f3d bytes) 000499[0] 1787364557.847602 mapwrite 0x13859d thru 0x142551 (0x9fb5 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4558 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x8c3c2d 000001[1] 1787364558.883136 trunc from 00000000 to 0x8c3c2c (0x8c3c2d bytes) truncating to largest ever: 0x9484b3 000002[1] 1787364558.951684 trunc from 0x8c3c2d to 0x9484b2 (0x84886 bytes) 000003[1] 1787364558.980610 write 0x07b000 thru 0x0817a3 (0x67a4 bytes) 000004[1] 1787364559.011392 trunc from 0x9484b3 to 0x729c19 (0x21e899 bytes) 000006[1] 1787364559.077135 mapwrite 0x1f3000 thru 0x1f6faf (0x3fb0 bytes) 000007[0] 1787364559.173124 mapwrite 0x60e000 thru 0x619ef5 (0xbef6 bytes) 000008[0] 1787364559.390030 mapread 0x4d2000 thru 0x4dbabc (0x9abd bytes) 000011[0] 1787364559.512772 mapread 0x2a6000 thru 0x2b0dd6 (0xadd7 bytes) 000012[1] 1787364559.568135 mapread 0x55a000 thru 0x566e9d (0xce9e bytes) 000013[1] 1787364559.668037 mapwrite 0x1e2000 thru 0x1f03f7 (0xe3f8 bytes) 000014[1] 1787364559.882471 trunc from 0x729c1a to 0x356d34 (0x3d2ee5 bytes) 000016[0] 1787364559.931392 mapwrite 0x0ad000 thru 0x0ae4a3 (0x14a4 bytes) 000017[1] 1787364560.145379 trunc from 0x356d35 to 0x23ebdd (0x118157 bytes) 000018[1] 1787364560.191732 read 0x1af000 thru 0x1b724c (0x824d bytes) 000020[1] 1787364560.236189 mapread 0x200000 thru 0x203691 (0x3692 bytes) 000025[0] 1787364560.288310 trunc from 0x23ebde to 0x80c136 (0x5cd559 bytes) 000026[0] 1787364560.322726 mapread 0x637000 thru 0x64429e (0xd29f bytes) 000029[0] 1787364560.376133 mapwrite 0x8e8000 thru 0x8eb3f9 (0x33fa bytes) 000035[0] 1787364560.537677 trunc from 0x8eb3fa to 0x23d1b6 (0x6ae243 bytes) 000037[1] 1787364560.602829 read 0x018000 thru 0x022739 (0xa73a bytes) 000040[1] 1787364560.714565 trunc from 0x23d1b7 to 0x039ad8 (0x2036de bytes) 000041[0] 1787364560.766446 read_OD 0x00a000 thru 0x00afff (0x1000 bytes) 000042[0] 1787364560.802323 mapread 0x00e000 thru 0x018bfc (0xabfd bytes) 000043[0] 1787364560.828686 mapwrite 0x3c5000 thru 0x3cc0f8 (0x70f9 bytes) 000045[1] 1787364561.049862 read 0x38a000 thru 0x3952a9 (0xb2aa bytes) 000046[0] 1787364561.154165 mapwrite 0x89b000 thru 0x89dd49 (0x2d4a bytes) 000054[0] 1787364561.416518 trunc from 0x89dd4a to 0x901c64 (0x63f1b bytes) 000055[1] 1787364561.449541 write 0x768000 thru 0x7705cc (0x85cd bytes) 000056[1] 1787364561.550484 mapread 0x068000 thru 0x0742a8 (0xc2a9 bytes) 000057[0] 1787364561.630972 read_OD 0x107000 thru 0x108fff (0x2000 bytes) 000058[1] 1787364561.670340 write 0x03b000 thru 0x0418d2 (0x68d3 bytes) 000059[1] 1787364561.702635 mapwrite 0x888000 thru 0x8886de (0x6df bytes) 000063[0] 1787364561.829874 trunc from 0x901c65 to 0x6e6e20 (0x21ae44 bytes) 000064[1] 1787364561.958665 read 0x178000 thru 0x184c46 (0xcc47 bytes) 000068[0] 1787364562.018558 trunc from 0x6e6e21 to 0x1bc178 (0x52aca8 bytes) 000069[0] 1787364562.057176 read_OD 0x101000 thru 0x103fff (0x3000 bytes) 000070[1] 1787364562.073840 mapwrite 0x5f6000 thru 0x5fdbd3 (0x7bd4 bytes) 000074[1] 1787364562.310420 write 0x59a000 thru 0x59c078 (0x2079 bytes) 000076[0] 1787364562.319850 mapread 0x5d7000 thru 0x5e3308 (0xc309 bytes) 000077[0] 1787364562.504649 read_OD 0x105000 thru 0x10ffff (0xb000 bytes) 000078[1] 1787364562.539884 mapwrite 0x570000 thru 0x578d95 (0x8d96 bytes) 000081[1] 1787364562.716376 mapread 0x524000 thru 0x52ebea (0xabeb bytes) 000085[0] 1787364562.776757 trunc from 0x5fdbd4 to 0x6568bf (0x58cec bytes) 000087[0] 1787364562.839576 trunc from 0x6568c0 to 0x1093a8 (0x54d517 bytes) 000091[0] 1787364562.866771 trunc from 0x1093a9 to 0x770fe2 (0x667c3a bytes) 000093[0] 1787364562.920532 write_OD 0x040000 thru 0x044fff (0x5000 bytes) 000095[0] 1787364563.008265 read_OD 0x333000 thru 0x33afff (0x8000 bytes) 000098[0] 1787364563.023556 trunc from 0x770fe3 to 0x56dccf (0x203313 bytes) 000099[1] 1787364563.077875 read 0x50d000 thru 0x519808 (0xc809 bytes) 000101[1] 1787364563.141084 mapwrite 0x26c000 thru 0x275b0b (0x9b0c bytes) 000102[0] 1787364563.270453 mapread 0x19d000 thru 0x19da85 (0xa86 bytes) 000104[1] 1787364563.307334 trunc from 0x56dcd0 to 0x06d065 (0x500c6a bytes) 000106[1] 1787364563.365616 write 0x371000 thru 0x37d5a0 (0xc5a1 bytes) 000112[0] 1787364563.371303 mapwrite 0x3c3000 thru 0x3d155e (0xe55f bytes) 000113[0] 1787364563.519573 mapread 0x2cf000 thru 0x2db519 (0xc51a bytes) 000117[1] 1787364563.625408 trunc from 0x3d155f to 0x896c06 (0x4c56a8 bytes) 000118[0] 1787364563.673104 write_OD 0x28b000 thru 0x28ffff (0x5000 bytes) 000119[1] 1787364563.763435 mapwrite 0x6f4000 thru 0x6fc34e (0x834f bytes) 000121[0] 1787364564.040258 read_OD 0x4dd000 thru 0x4e5fff (0x9000 bytes) 000122[1] 1787364564.082849 mapwrite 0x4eb000 thru 0x4f91c3 (0xe1c4 bytes) 000124[1] 1787364564.230975 read 0x698000 thru 0x69e0c9 (0x60ca bytes) 000125[0] 1787364564.243776 mapwrite 0x09a000 thru 0x09a92a (0x92b bytes) 000126[0] 1787364564.434472 mapwrite 0x126000 thru 0x13543b (0xf43c bytes) 000130[1] 1787364564.524024 mapread 0x5e6000 thru 0x5e7bdf (0x1be0 bytes) 000131[0] 1787364564.560853 write_OD 0x60e000 thru 0x61afff (0xd000 bytes) 000132[1] 1787364564.643376 mapread 0x0db000 thru 0x0dcc40 (0x1c41 bytes) 000133[0] 1787364564.664295 mapread 0x168000 thru 0x16c57d (0x457e bytes) 000135[1] 1787364564.706184 mapwrite 0x437000 thru 0x4406a5 (0x96a6 bytes) 000136[0] 1787364564.836659 write_OD 0x96d000 thru 0x973fff (0x7000 bytes) 000137[0] 1787364564.900306 trunc from 0x974000 to 0x6a5900 (0x2ce6ff bytes) 000139[0] 1787364564.940923 mapwrite 0x262000 thru 0x26a1f7 (0x81f8 bytes) 000141[0] 1787364565.078759 write_OD 0x470000 thru 0x478fff (0x9000 bytes) 000143[1] 1787364565.185317 mapread 0x2b3000 thru 0x2c2717 (0xf718 bytes) 000144[0] 1787364565.218237 read_OD 0x177000 thru 0x185fff (0xf000 bytes) 000147[0] 1787364565.232299 trunc from 0x6a5901 to 0x7a983f (0x103f3f bytes) 000148[0] 1787364565.276762 trunc from 0x7a9840 to 0x72113a (0x88705 bytes) 000149[1] 1787364565.288852 read 0x4e7000 thru 0x4ebff9 (0x4ffa bytes) 000151[1] 1787364565.314485 write 0x275000 thru 0x27dc5b (0x8c5c bytes) 000152[1] 1787364565.351452 mapwrite 0x927000 thru 0x92a3c8 (0x33c9 bytes) 000156[0] 1787364565.462935 write_OD 0x68c000 thru 0x695fff (0xa000 bytes) 000159[1] 1787364565.531743 read 0x70f000 thru 0x714166 (0x5167 bytes) 000160[0] 1787364565.548817 write_OD 0x9e8000 thru 0x9eefff (0x7000 bytes) 000161[0] 1787364565.641683 read_OD 0x35f000 thru 0x361fff (0x3000 bytes) 000162[1] 1787364565.652785 read 0x046000 thru 0x04a7ca (0x47cb bytes) 000164[0] 1787364565.664490 mapread 0x75c000 thru 0x763565 (0x7566 bytes) 000167[0] 1787364565.691754 write_OD 0x9c6000 thru 0x9cbfff (0x6000 bytes) 000169[1] 1787364565.821384 mapwrite 0x9c9000 thru 0x9cdfa5 (0x4fa6 bytes) 000170[1] 1787364566.009362 write 0x6b2000 thru 0x6b9d91 (0x7d92 bytes) 000171[1] 1787364566.032959 mapwrite 0x273000 thru 0x27a297 (0x7298 bytes) 000172[1] 1787364566.104665 trunc from 0x9ef000 to 0x8828f4 (0x16c70b bytes) 000173[1] 1787364566.194312 mapread 0x879000 thru 0x87cdd9 (0x3dda bytes) 000174[0] 1787364566.256186 mapwrite 0x0c3000 thru 0x0d2ab0 (0xfab1 bytes) 000179[0] 1787364566.502167 read_OD 0x5fc000 thru 0x601fff (0x6000 bytes) 000184[0] 1787364566.526144 read_OD 0x53d000 thru 0x54bfff (0xf000 bytes) 000186[0] 1787364566.537184 write_OD 0x07f000 thru 0x087fff (0x9000 bytes) 000187[1] 1787364566.598373 write 0x40a000 thru 0x41945f (0xf460 bytes) 000188[0] 1787364566.645481 mapwrite 0x34c000 thru 0x356e01 (0xae02 bytes) 000189[0] 1787364566.887144 write_OD 0x398000 thru 0x3a5fff (0xe000 bytes) 000191[1] 1787364566.968076 mapwrite 0x459000 thru 0x4613a3 (0x83a4 bytes) 000192[0] 1787364567.184271 mapread 0x525000 thru 0x5294cc (0x44cd bytes) 000193[0] 1787364567.259697 mapwrite 0x5a7000 thru 0x5aad7c (0x3d7d bytes) 000194[0] 1787364567.327104 read_OD 0x2da000 thru 0x2e8fff (0xf000 bytes) 000195[0] 1787364567.347884 mapwrite 0x353000 thru 0x358494 (0x5495 bytes) 000198[1] 1787364567.463388 read 0x0ca000 thru 0x0cca18 (0x2a19 bytes) 000199[1] 1787364567.514611 read 0x214000 thru 0x222cc6 (0xecc7 bytes) 000200[0] 1787364567.545765 trunc from 0x8828f5 to 0x542aee (0x33fe06 bytes) truncating to largest ever: 0x9bdb84 000202[0] 1787364567.607705 trunc from 0x542aef to 0x9bdb83 (0x47b095 bytes) 000204[1] 1787364567.651622 write 0x784000 thru 0x78a94d (0x694e bytes) 000206[1] 1787364567.685149 trunc from 0x9bdb84 to 0x602524 (0x3bb65f bytes) 000208[0] 1787364567.718772 read_OD 0x567000 thru 0x571fff (0xb000 bytes) 000210[1] 1787364567.738424 write 0x3ff000 thru 0x40059d (0x159e bytes) 000211[1] 1787364567.753053 trunc from 0x602525 to 0x925788 (0x323264 bytes) 000213[1] 1787364567.797768 trunc from 0x925789 to 0x89abb9 (0x8abcf bytes) 000215[0] 1787364567.845881 read_OD 0x1e9000 thru 0x1f4fff (0xc000 bytes) 000217[0] 1787364567.896243 write_OD 0x0db000 thru 0x0e4fff (0xa000 bytes) 000221[1] 1787364567.967494 write 0x5bd000 thru 0x5c042a (0x342b bytes) 000224[0] 1787364568.030175 mapwrite 0x041000 thru 0x047063 (0x6064 bytes) 000225[1] 1787364568.129476 read 0x432000 thru 0x43ddd9 (0xbdda bytes) 000227[0] 1787364568.164290 mapwrite 0x103000 thru 0x10ddae (0xadaf bytes) 000230[1] 1787364568.341461 mapwrite 0x2ce000 thru 0x2db087 (0xd088 bytes) 000232[1] 1787364568.439111 mapread 0x461000 thru 0x46cf0c (0xbf0d bytes) 000234[1] 1787364568.481689 write 0x200000 thru 0x200ba1 (0xba2 bytes) 000236[0] 1787364568.517677 mapread 0x112000 thru 0x11ec4b (0xcc4c bytes) 000238[1] 1787364568.564833 read 0x470000 thru 0x47151f (0x1520 bytes) 000239[0] 1787364568.579907 trunc from 0x89abba to 0x08074d (0x81a46c bytes) 000242[0] 1787364568.661347 mapread 0x02b000 thru 0x02c5b8 (0x15b9 bytes) 000243[0] 1787364568.700845 trunc from 0x08074e to 0x23adda (0x1ba68d bytes) 000245[0] 1787364568.787737 read_OD 0x002000 thru 0x00dfff (0xc000 bytes) 000246[0] 1787364568.820314 mapread 0x22f000 thru 0x2307bb (0x17bc bytes) 000248[0] 1787364568.870623 write_OD 0x6de000 thru 0x6e3fff (0x6000 bytes) 000249[1] 1787364568.961696 write 0x361000 thru 0x36475e (0x375f bytes) 000253[0] 1787364569.033997 mapread 0x144000 thru 0x14743a (0x343b bytes) 000254[0] 1787364569.182107 trunc from 0x6e4000 to 0x2798ca (0x46a735 bytes) 000255[1] 1787364569.223124 mapwrite 0x0bb000 thru 0x0c7e24 (0xce25 bytes) 000258[0] 1787364569.369667 mapwrite 0x1db000 thru 0x1e0f78 (0x5f79 bytes) 000260[0] 1787364569.475710 trunc from 0x2798cb to 0x0c5088 (0x1b4842 bytes) 000261[0] 1787364569.515766 read_OD 0x016000 thru 0x018fff (0x3000 bytes) 000262[1] 1787364569.535992 read 0x016000 thru 0x01a4db (0x44dc bytes) 000263[0] 1787364569.564318 read_OD 0x0b8000 thru 0x0c1fff (0xa000 bytes) 000264[1] 1787364569.573368 trunc from 0x0c5089 to 0x4fa4a8 (0x435420 bytes) 000266[1] 1787364569.642332 mapwrite 0x32d000 thru 0x32f1f2 (0x21f3 bytes) 000268[1] 1787364569.805452 read 0x00e000 thru 0x0135db (0x55dc bytes) 000269[0] 1787364569.829912 read_OD 0x003000 thru 0x00dfff (0xb000 bytes) 000274[0] 1787364569.878703 mapread 0x25b000 thru 0x2617f4 (0x67f5 bytes) 000276[0] 1787364569.900902 trunc from 0x4fa4a9 to 0x9406ad (0x446205 bytes) 000277[1] 1787364569.931746 trunc from 0x9406ae to 0x951c44 (0x11597 bytes) 000278[0] 1787364569.981639 write_OD 0x2e2000 thru 0x2e4fff (0x3000 bytes) 000279[0] 1787364570.034390 write_OD 0x694000 thru 0x69ffff (0xc000 bytes) 000281[0] 1787364570.095945 mapwrite 0x9bd000 thru 0x9c5449 (0x844a bytes) 000282[0] 1787364570.214268 mapwrite 0x8c8000 thru 0x8cfd87 (0x7d88 bytes) 000283[0] 1787364570.279447 mapwrite 0x452000 thru 0x45adca (0x8dcb bytes) 000284[0] 1787364570.432787 write_OD 0x7bf000 thru 0x7c1fff (0x3000 bytes) 000286[0] 1787364570.534858 read_OD 0x786000 thru 0x78efff (0x9000 bytes) 000287[0] 1787364570.551175 trunc from 0x9c544a to 0x526e22 (0x49e627 bytes) 000289[1] 1787364570.626675 read 0x403000 thru 0x409ae8 (0x6ae9 bytes) 000292[0] 1787364570.697829 trunc from 0x526e23 to 0x8dbbf6 (0x3b4dd4 bytes) 000293[0] 1787364570.796585 read_OD 0x018000 thru 0x01afff (0x3000 bytes) 000295[0] 1787364570.834147 write_OD 0x50b000 thru 0x518fff (0xe000 bytes) 000299[0] 1787364570.945947 read_OD 0x538000 thru 0x53dfff (0x6000 bytes) 000301[1] 1787364570.958624 write 0x888000 thru 0x892677 (0xa678 bytes) 000303[0] 1787364571.001304 mapread 0x410000 thru 0x41ecea (0xeceb bytes) 000304[1] 1787364571.069347 read 0x243000 thru 0x24b995 (0x8996 bytes) 000305[1] 1787364571.086110 mapwrite 0x997000 thru 0x9a523e (0xe23f bytes) 000306[0] 1787364571.212695 mapread 0x844000 thru 0x848bd8 (0x4bd9 bytes) 000307[1] 1787364571.261885 trunc from 0x9a523f to 0x5a5c92 (0x3ff5ac bytes) 000308[0] 1787364571.314322 write_OD 0x03e000 thru 0x04cfff (0xf000 bytes) 000309[1] 1787364571.374759 read 0x3ab000 thru 0x3abdf5 (0xdf6 bytes) 000310[0] 1787364571.388128 mapwrite 0x511000 thru 0x51d854 (0xc855 bytes) 000311[0] 1787364571.478438 write_OD 0x413000 thru 0x416fff (0x4000 bytes) 000313[0] 1787364571.525112 mapread 0x2f7000 thru 0x2fd509 (0x650a bytes) 000314[1] 1787364571.545280 trunc from 0x5a5c93 to 0x01c5d6 (0x5896bc bytes) 000316[1] 1787364571.602586 trunc from 0x01c5d7 to 0x1b1068 (0x194a92 bytes) 000318[1] 1787364571.635167 mapwrite 0x5fa000 thru 0x6087e4 (0xe7e5 bytes) 000319[0] 1787364571.730887 write_OD 0x824000 thru 0x825fff (0x2000 bytes) 000323[1] 1787364571.790670 trunc from 0x826000 to 0x146b84 (0x6df47b bytes) 000325[0] 1787364571.846503 mapread 0x0bb000 thru 0x0c16bb (0x66bc bytes) 000329[1] 1787364571.894893 mapread 0x137000 thru 0x14022e (0x922f bytes) 000330[0] 1787364571.920002 mapread 0x06a000 thru 0x0726d1 (0x86d2 bytes) 000331[1] 1787364571.932377 trunc from 0x146b85 to 0x7a08b9 (0x659d35 bytes) 000334[0] 1787364571.990471 write_OD 0x28c000 thru 0x28dfff (0x2000 bytes) 000335[0] 1787364572.035173 mapread 0x322000 thru 0x322924 (0x925 bytes) 000336[0] 1787364572.046500 mapwrite 0x0cc000 thru 0x0da40c (0xe40d bytes) 000338[1] 1787364572.164923 trunc from 0x7a08ba to 0x8c743e (0x126b85 bytes) 000340[0] 1787364572.219949 write_OD 0x005000 thru 0x00afff (0x6000 bytes) 000341[1] 1787364572.269962 trunc from 0x8c743f to 0x7ac703 (0x11ad3b bytes) 000343[1] 1787364572.332430 mapwrite 0x3dd000 thru 0x3e2df2 (0x5df3 bytes) 000345[0] 1787364572.413393 trunc from 0x7ac704 to 0x21cdcf (0x58f934 bytes) 000348[0] 1787364572.455749 trunc from 0x21cdd0 to 0x525dea (0x30901b bytes) 000349[1] 1787364572.486493 read 0x1dc000 thru 0x1e55f0 (0x95f1 bytes) 000352[0] 1787364572.517299 mapwrite 0x54d000 thru 0x552acb (0x5acc bytes) 000355[1] 1787364572.625103 mapwrite 0x0d9000 thru 0x0e5867 (0xc868 bytes) 000356[0] 1787364572.724761 read_OD 0x474000 thru 0x480fff (0xd000 bytes) 000359[1] 1787364572.757277 trunc from 0x552acc to 0x69ea0b (0x14bf40 bytes) 000360[0] 1787364572.790640 trunc from 0x69ea0c to 0x781090 (0xe2685 bytes) 000361[0] 1787364572.825459 mapwrite 0x12a000 thru 0x137bb3 (0xdbb4 bytes) 000363[1] 1787364572.914178 mapwrite 0x8e7000 thru 0x8e91d6 (0x21d7 bytes) 000365[1] 1787364573.040778 trunc from 0x8e91d7 to 0x6a7433 (0x241da3 bytes) 000370[0] 1787364573.083964 read_OD 0x10f000 thru 0x116fff (0x8000 bytes) 000374[0] 1787364573.118954 mapread 0x266000 thru 0x2694d1 (0x34d2 bytes) 000378[0] 1787364573.139603 mapread 0x3cf000 thru 0x3d390b (0x490c bytes) 000381[1] 1787364573.168650 write 0x6ce000 thru 0x6d054a (0x254b bytes) 000383[0] 1787364573.245805 read_OD 0x36a000 thru 0x373fff (0xa000 bytes) 000384[0] 1787364573.307844 mapread 0x090000 thru 0x09b71e (0xb71f bytes) 000387[1] 1787364573.337133 read 0x400000 thru 0x4094ce (0x94cf bytes) 000389[0] 1787364573.344966 trunc from 0x6d054b to 0x322222 (0x3ae328 bytes) 000390[1] 1787364573.383676 mapread 0x076000 thru 0x08188f (0xb890 bytes) 000391[0] 1787364573.438457 mapread 0x30c000 thru 0x30f706 (0x3707 bytes) 000393[1] 1787364573.495110 mapread 0x05c000 thru 0x06bbbc (0xfbbd bytes) 000395[0] 1787364573.519349 mapwrite 0x7da000 thru 0x7e95d5 (0xf5d6 bytes) 000396[1] 1787364573.661330 read 0x66e000 thru 0x67bf65 (0xdf66 bytes) 000399[1] 1787364573.686261 mapread 0x4bf000 thru 0x4cd0eb (0xe0ec bytes) 000400[1] 1787364573.712524 mapread 0x6bb000 thru 0x6c4ba7 (0x9ba8 bytes) 000402[0] 1787364573.742166 mapread 0x74c000 thru 0x753546 (0x7547 bytes) 000403[0] 1787364573.752726 mapwrite 0x11f000 thru 0x124bbf (0x5bc0 bytes) 000404[1] 1787364573.853367 mapwrite 0x313000 thru 0x315e09 (0x2e0a bytes) 000405[1] 1787364573.976515 write 0x47f000 thru 0x487b18 (0x8b19 bytes) 000406[0] 1787364573.989352 mapread 0x629000 thru 0x63395d (0xa95e bytes) 000408[1] 1787364574.036650 trunc from 0x7e95d6 to 0x417527 (0x3d20ae bytes) 000409[1] 1787364574.086812 mapwrite 0x9d8000 thru 0x9e5d7c (0xdd7d bytes) 000413[1] 1787364574.249761 read 0x420000 thru 0x42df58 (0xdf59 bytes) 000414[0] 1787364574.265286 mapread 0x286000 thru 0x2952a0 (0xf2a1 bytes) 000416[0] 1787364574.321380 trunc from 0x9e5d7d to 0x958f0f (0x8ce6d bytes) 000417[0] 1787364574.354087 write_OD 0x290000 thru 0x29bfff (0xc000 bytes) 000418[1] 1787364574.475276 mapread 0x297000 thru 0x29c779 (0x577a bytes) 000419[1] 1787364574.505485 mapwrite 0x0d4000 thru 0x0d9634 (0x5635 bytes) 000423[1] 1787364574.633879 trunc from 0x958f10 to 0x1bcdaf (0x79c160 bytes) 000429[1] 1787364574.678803 mapwrite 0x332000 thru 0x33eb9d (0xcb9e bytes) 000430[1] 1787364574.777873 trunc from 0x33eb9e to 0x3dacfa (0x9c15d bytes) 000433[1] 1787364574.830588 mapwrite 0x2fa000 thru 0x30292b (0x892c bytes) 000436[1] 1787364574.924173 read 0x380000 thru 0x38a265 (0xa266 bytes) 000437[1] 1787364575.004269 mapread 0x398000 thru 0x3a05ea (0x85eb bytes) 000439[1] 1787364575.021958 mapwrite 0x884000 thru 0x891334 (0xd335 bytes) 000440[0] 1787364575.303836 trunc from 0x891335 to 0x78f416 (0x101f1e bytes) 000444[0] 1787364575.411412 read_OD 0x42d000 thru 0x438fff (0xc000 bytes) 000446[1] 1787364575.441723 read 0x777000 thru 0x7810dc (0xa0dd bytes) 000447[0] 1787364575.454872 mapread 0x6ce000 thru 0x6ce4aa (0x4ab bytes) 000449[0] 1787364575.460918 trunc from 0x78f417 to 0x334264 (0x45b1b2 bytes) 000451[0] 1787364575.520627 mapread 0x1a7000 thru 0x1a7bee (0xbef bytes) 000453[0] 1787364575.545315 trunc from 0x334265 to 0x63f777 (0x30b513 bytes) 000454[0] 1787364575.574721 mapread 0x4de000 thru 0x4eaf4b (0xcf4c bytes) 000455[1] 1787364575.622379 write 0x8c6000 thru 0x8caad7 (0x4ad8 bytes) 000458[0] 1787364575.681328 mapread 0x86e000 thru 0x87305b (0x505c bytes) 000460[0] 1787364575.737843 trunc from 0x8caad8 to 0x01c6c7 (0x8ae410 bytes) 000461[0] 1787364575.773863 write_OD 0x3e8000 thru 0x3e8fff (0x1000 bytes) 000468[1] 1787364575.859660 write 0x25b000 thru 0x267112 (0xc113 bytes) 000471[0] 1787364575.906925 write_OD 0x487000 thru 0x48bfff (0x5000 bytes) 000472[1] 1787364575.991302 mapwrite 0x4ca000 thru 0x4d7508 (0xd509 bytes) 000476[1] 1787364576.162302 mapwrite 0x669000 thru 0x66f725 (0x6726 bytes) 000478[1] 1787364576.272274 mapwrite 0x198000 thru 0x19aa52 (0x2a53 bytes) 000479[1] 1787364576.332268 mapwrite 0x2a3000 thru 0x2aa9e1 (0x79e2 bytes) 000480[1] 1787364576.403406 mapread 0x279000 thru 0x27cfa5 (0x3fa6 bytes) 000481[0] 1787364576.413423 write_OD 0x3ad000 thru 0x3b9fff (0xd000 bytes) 000483[1] 1787364576.469123 write 0x587000 thru 0x5942c8 (0xd2c9 bytes) 000488[0] 1787364576.531585 write_OD 0x74f000 thru 0x753fff (0x5000 bytes) 000489[0] 1787364576.595685 mapread 0x583000 thru 0x58c898 (0x9899 bytes) 000490[1] 1787364576.616852 mapwrite 0x291000 thru 0x294322 (0x3323 bytes) 000496[1] 1787364576.739904 mapread 0x70d000 thru 0x7158aa (0x88ab bytes) 000499[0] 1787364576.777707 trunc from 0x754000 to 0x4feead (0x255152 bytes) All operations completed A-OK! PASS 16a (58s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 22:09:42 (1787364582) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.360688 s, 23.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.416251 s, 20.2 MB/s Chance of close/open is 1 in 50 Seed set to 4585 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1787364585.876692 mapwrite 0x577843 thru 0x578763 (0xf21 bytes) 000003[1] 1787364585.970956 mapwrite 0x0a85c3 thru 0x0a9366 (0xda4 bytes) 000005[1] 1787364586.048900 mapread 0x037e0e thru 0x037f13 (0x106 bytes) 000006[1] 1787364586.061573 mapwrite 0x097223 thru 0x0972af (0x8d bytes) 000007[1] 1787364586.105531 mapread 0x45b871 thru 0x45c1d7 (0x967 bytes) 000008[1] 1787364586.144960 write 0x9d4330 thru 0x9d4b90 (0x861 bytes) 000010[1] 1787364586.200814 read 0x56ca2e thru 0x56dc5c (0x122f bytes) 000011[0] 1787364586.215492 mapwrite 0x4305e9 thru 0x430b27 (0x53f bytes) truncating to largest ever: 0x6922be 000012[0] 1787364586.452407 trunc from 0x9d4b91 to 0x6922bd (0x3428d3 bytes) 000013[0] 1787364586.567789 mapwrite 0x0bd50c thru 0x0be7ae (0x12a3 bytes) truncating to largest ever: 0x98370f 000014[1] 1787364586.723929 trunc from 0x6922be to 0x98370e (0x2f1451 bytes) 000015[1] 1787364586.853471 mapread 0x2f126f thru 0x2f25ac (0x133e bytes) 000016[1] 1787364586.897954 mapread 0x7125c5 thru 0x713604 (0x1040 bytes) 000017[0] 1787364586.938879 mapread 0x6d6a81 thru 0x6d7e53 (0x13d3 bytes) 000022[0] 1787364587.039911 read 0x7d789e thru 0x7d7b91 (0x2f4 bytes) 000026[1] 1787364587.085629 mapwrite 0x3e5cf4 thru 0x3e6514 (0x821 bytes) 000028[1] 1787364587.147794 mapwrite 0x4480f5 thru 0x44984b (0x1757 bytes) 000037[1] 1787364587.345755 trunc from 0x98370f to 0x58ccf5 (0x3f6a19 bytes) 000038[0] 1787364587.400658 write 0x4d5fa8 thru 0x4d63cb (0x424 bytes) 000039[0] 1787364587.439685 mapread 0x03a297 thru 0x03a641 (0x3ab bytes) 000040[0] 1787364587.489125 mapwrite 0x527192 thru 0x528a94 (0x1903 bytes) 000047[1] 1787364587.639962 read 0x394aef thru 0x39543c (0x94e bytes) 000049[1] 1787364587.665524 mapread 0x4400f1 thru 0x441528 (0x1438 bytes) 000050[0] 1787364587.703672 trunc from 0x58ccf6 to 0x23dc1a (0x34f0db bytes) 000052[0] 1787364587.778600 write 0x6958fe thru 0x695ad1 (0x1d4 bytes) 000054[1] 1787364587.866719 mapwrite 0x9f0da4 thru 0x9f1b51 (0xdae bytes) 000055[0] 1787364588.030854 mapwrite 0x95e2ea thru 0x95fadc (0x17f3 bytes) 000056[0] 1787364588.188072 write 0x7ce663 thru 0x7cf138 (0xad6 bytes) 000057[0] 1787364588.244356 mapwrite 0x957ac5 thru 0x957b51 (0x8d bytes) 000059[0] 1787364588.341634 write 0x7ce591 thru 0x7cef16 (0x986 bytes) 000060[1] 1787364588.342439 mapwrite 0x6462c5 thru 0x646585 (0x2c1 bytes) 000061[1] 1787364588.432633 mapwrite 0x2a1050 thru 0x2a1b47 (0xaf8 bytes) 000062[1] 1787364588.511509 trunc from 0x9f1b52 to 0x68b0a9 (0x366aa8 bytes) 000063[1] 1787364588.536335 trunc from 0x68b0aa to 0x732c73 (0xa7bca bytes) 000065[0] 1787364588.576626 mapread 0x42656c thru 0x426b2d (0x5c2 bytes) 000066[1] 1787364588.627718 mapwrite 0x1e2f5d thru 0x1e36c9 (0x76d bytes) 000067[0] 1787364588.726479 read 0x486a03 thru 0x487368 (0x966 bytes) 000069[0] 1787364588.782829 trunc from 0x732c74 to 0x491ae1 (0x2a1192 bytes) 000070[0] 1787364588.861766 trunc from 0x491ae2 to 0x3bd49b (0xd4646 bytes) 000071[1] 1787364588.886031 mapwrite 0x200f54 thru 0x201d6f (0xe1c bytes) 000072[1] 1787364589.077281 write 0x9b3182 thru 0x9b41db (0x105a bytes) 000074[0] 1787364589.151719 mapwrite 0x2f2e43 thru 0x2f377d (0x93b bytes) 000076[0] 1787364589.392128 read 0x15e949 thru 0x160144 (0x17fc bytes) 000077[1] 1787364589.398162 read 0x60ef77 thru 0x610880 (0x190a bytes) 000078[0] 1787364589.440260 mapread 0x2bd973 thru 0x2bf556 (0x1be4 bytes) 000080[1] 1787364589.474881 read 0x77e717 thru 0x77ec06 (0x4f0 bytes) 000082[1] 1787364589.490017 trunc from 0x9b41dc to 0x7064e8 (0x2adcf3 bytes) 000085[1] 1787364589.630143 trunc from 0x7064e9 to 0x396bac (0x36f93c bytes) 000092[0] 1787364589.863445 write 0x6d3d94 thru 0x6d3dbc (0x29 bytes) 000095[0] 1787364589.920137 mapwrite 0x2a643d thru 0x2a6a61 (0x625 bytes) 000097[1] 1787364589.986502 write 0x55714f thru 0x5587c5 (0x1677 bytes) 000098[0] 1787364590.024510 mapread 0x1db4de thru 0x1dc5f7 (0x111a bytes) 000100[0] 1787364590.076307 write 0x6756a4 thru 0x6764d6 (0xe33 bytes) 000103[0] 1787364590.124487 trunc from 0x6d3dbd to 0x9744cf (0x2a0713 bytes) 000105[1] 1787364590.147537 trunc from 0x9744d0 to 0x2c52a5 (0x6af22a bytes) 000106[0] 1787364590.225600 read 0x219a1e thru 0x219dac (0x38f bytes) 000107[0] 1787364590.254848 read 0x131e44 thru 0x133cd8 (0x1e95 bytes) 000108[0] 1787364590.325546 read 0x1702b9 thru 0x17197f (0x16c7 bytes) 000112[0] 1787364590.385326 mapwrite 0x1f9e5d thru 0x1fb302 (0x14a6 bytes) 000114[0] 1787364590.597371 write 0x36d1bf thru 0x36e65a (0x149c bytes) 000115[0] 1787364590.651362 mapwrite 0x8a8e04 thru 0x8a98db (0xad8 bytes) 000117[0] 1787364590.882942 read 0x38f825 thru 0x3916e4 (0x1ec0 bytes) 000118[1] 1787364590.883518 trunc from 0x8a98dc to 0x248119 (0x6617c2 bytes) 000119[0] 1787364590.946518 write 0x0bdaa3 thru 0x0be6de (0xc3c bytes) 000120[1] 1787364590.977481 mapwrite 0x2ced06 thru 0x2d04b6 (0x17b1 bytes) 000122[0] 1787364591.108958 read 0x16abd8 thru 0x16b5fe (0xa27 bytes) 000124[0] 1787364591.233040 read 0x1fd4ee thru 0x1fdc37 (0x74a bytes) 000125[1] 1787364591.234288 mapwrite 0x050008 thru 0x05195b (0x1954 bytes) 000126[1] 1787364591.312368 write 0x0adadd thru 0x0af218 (0x173c bytes) 000127[0] 1787364591.347777 write 0x867510 thru 0x867c98 (0x789 bytes) 000129[0] 1787364591.396702 trunc from 0x867c99 to 0x189326 (0x6de972 bytes) 000131[1] 1787364591.455833 trunc from 0x189327 to 0x3112a5 (0x187f7f bytes) 000134[0] 1787364591.547694 trunc from 0x3112a6 to 0x932b5f (0x6218ba bytes) 000135[1] 1787364591.603796 mapwrite 0x7086ea thru 0x70a07c (0x1993 bytes) 000137[1] 1787364591.746186 read 0x583868 thru 0x5842e5 (0xa7e bytes) 000139[1] 1787364591.797323 mapwrite 0x2aa627 thru 0x2abf90 (0x196a bytes) 000141[1] 1787364591.942284 write 0x6b1366 thru 0x6b2b93 (0x182e bytes) 000142[0] 1787364591.952807 mapwrite 0x014eb7 thru 0x015068 (0x1b2 bytes) 000145[1] 1787364592.165917 mapread 0x64250e thru 0x6441bd (0x1cb0 bytes) 000146[1] 1787364592.213443 trunc from 0x932b60 to 0x362f23 (0x5cfc3c bytes) 000151[1] 1787364592.320705 write 0x7cad13 thru 0x7cc8d7 (0x1bc5 bytes) 000152[0] 1787364592.326438 mapwrite 0x48207f thru 0x4822fe (0x280 bytes) truncating to largest ever: 0x9b510f 000156[1] 1787364592.505960 trunc from 0x7cc8d8 to 0x9b510e (0x1e8837 bytes) 000157[0] 1787364592.561672 mapread 0x3d5e1f thru 0x3d6264 (0x446 bytes) 000158[0] 1787364592.607897 mapwrite 0x9bd961 thru 0x9bef9c (0x163c bytes) 000159[1] 1787364592.728040 trunc from 0x9bef9d to 0x5fff49 (0x3bf053 bytes) 000161[0] 1787364592.836257 mapwrite 0x8b3f50 thru 0x8b4d0d (0xdbe bytes) truncating to largest ever: 0x9dbfa2 000162[0] 1787364592.921283 trunc from 0x8b4d0e to 0x9dbfa1 (0x127294 bytes) 000163[1] 1787364592.944980 mapread 0x82a15f thru 0x82b2c5 (0x1167 bytes) 000165[0] 1787364592.984652 mapwrite 0x181068 thru 0x182fa2 (0x1f3b bytes) 000168[0] 1787364593.083776 trunc from 0x9dbfa2 to 0x5ea1fb (0x3f1da6 bytes) 000169[0] 1787364593.128925 mapread 0x300c07 thru 0x301aa5 (0xe9f bytes) 000172[1] 1787364593.145754 trunc from 0x5ea1fc to 0x425aea (0x1c4711 bytes) 000174[1] 1787364593.222156 mapread 0x187bbc thru 0x188c4d (0x1092 bytes) 000175[0] 1787364593.232513 write 0x201c11 thru 0x201c59 (0x49 bytes) 000177[1] 1787364593.273513 write 0x3066ed thru 0x3074e4 (0xdf8 bytes) 000180[1] 1787364593.301932 write 0x162e60 thru 0x163e64 (0x1005 bytes) 000181[0] 1787364593.314302 mapread 0x095672 thru 0x096038 (0x9c7 bytes) 000182[1] 1787364593.367547 trunc from 0x425aeb to 0x7f5419 (0x3cf92f bytes) 000183[1] 1787364593.407686 read 0x01fec4 thru 0x021e07 (0x1f44 bytes) 000188[1] 1787364593.465436 mapwrite 0x1c644c thru 0x1c81e8 (0x1d9d bytes) 000191[0] 1787364593.604377 read 0x45baff thru 0x45d128 (0x162a bytes) 000192[0] 1787364593.630915 mapread 0x077537 thru 0x078dbc (0x1886 bytes) 000193[1] 1787364593.669107 write 0x979526 thru 0x9798a5 (0x380 bytes) 000194[1] 1787364593.685812 mapread 0x840f67 thru 0x841ee0 (0xf7a bytes) 000198[1] 1787364593.728652 mapread 0x74f70d thru 0x750cc7 (0x15bb bytes) 000202[0] 1787364593.779419 write 0x3fc964 thru 0x3fd24a (0x8e7 bytes) 000203[1] 1787364593.817868 mapwrite 0x0922df thru 0x093fc9 (0x1ceb bytes) 000204[0] 1787364593.905855 mapread 0x8f29b4 thru 0x8f4131 (0x177e bytes) 000205[1] 1787364593.942967 mapwrite 0x02f6a7 thru 0x02fad3 (0x42d bytes) 000206[1] 1787364594.020111 read 0x040e19 thru 0x041ab1 (0xc99 bytes) 000208[1] 1787364594.049951 mapread 0x019ede thru 0x01b2e2 (0x1405 bytes) 000209[1] 1787364594.052726 write 0x003783 thru 0x003d3b (0x5b9 bytes) 000214[0] 1787364594.069125 read 0x1311bd thru 0x131271 (0xb5 bytes) 000223[1] 1787364594.098511 trunc from 0x9798a6 to 0x24a739 (0x72f16c bytes) 000224[1] 1787364594.168907 mapwrite 0x9091bf thru 0x9097a6 (0x5e8 bytes) 000226[1] 1787364594.265124 read 0x78c8a0 thru 0x78da80 (0x11e1 bytes) 000228[0] 1787364594.271444 read 0x878081 thru 0x878385 (0x305 bytes) 000230[1] 1787364594.323646 trunc from 0x9097a7 to 0x9ab49b (0xa1cf5 bytes) 000231[1] 1787364594.410944 trunc from 0x9ab49c to 0x3f6f63 (0x5b4538 bytes) 000233[0] 1787364594.457004 mapwrite 0x557039 thru 0x5584df (0x14a7 bytes) 000234[0] 1787364594.729969 mapread 0x1a9442 thru 0x1aa26e (0xe2d bytes) 000235[0] 1787364594.773972 write 0x8557cf thru 0x856eed (0x171f bytes) 000236[0] 1787364594.778534 mapread 0x2e432b thru 0x2e4669 (0x33f bytes) 000237[1] 1787364594.795515 write 0x088945 thru 0x08941a (0xad6 bytes) 000238[0] 1787364594.847364 mapwrite 0x87fb26 thru 0x8806c2 (0xb9d bytes) 000242[1] 1787364594.946620 mapwrite 0x726ee1 thru 0x72739b (0x4bb bytes) 000243[1] 1787364595.034527 read 0x4c1da1 thru 0x4c3bcc (0x1e2c bytes) 000245[1] 1787364595.070121 trunc from 0x8806c3 to 0x451acc (0x42ebf6 bytes) 000246[1] 1787364595.100845 mapwrite 0x2c4de8 thru 0x2c6417 (0x1630 bytes) 000247[0] 1787364595.185548 mapread 0x02085b thru 0x021211 (0x9b7 bytes) 000248[0] 1787364595.288890 write 0x32050c thru 0x32073b (0x230 bytes) 000250[0] 1787364595.372960 read 0x395dd5 thru 0x396ffa (0x1226 bytes) 000254[0] 1787364595.443161 trunc from 0x451acd to 0x4c5ef8 (0x7442c bytes) 000255[0] 1787364595.494369 mapread 0x10fd85 thru 0x111886 (0x1b02 bytes) 000258[1] 1787364595.564745 write 0x7cddbf thru 0x7cfc6c (0x1eae bytes) 000262[1] 1787364595.649526 write 0x0b9090 thru 0x0ba246 (0x11b7 bytes) 000265[1] 1787364595.702520 write 0x0be1fb thru 0x0bf5d5 (0x13db bytes) 000266[1] 1787364595.725486 trunc from 0x7cfc6d to 0x87d2ff (0xad693 bytes) 000268[0] 1787364595.814526 trunc from 0x87d300 to 0x897ccc (0x1a9cd bytes) 000272[0] 1787364595.904026 mapread 0x20b3c8 thru 0x20c1c2 (0xdfb bytes) 000276[0] 1787364595.958508 write 0x316163 thru 0x3165f0 (0x48e bytes) 000279[0] 1787364595.985726 mapwrite 0x545a6c thru 0x546f12 (0x14a7 bytes) 000286[1] 1787364596.081479 mapread 0x0652d5 thru 0x065b69 (0x895 bytes) 000290[1] 1787364596.160872 write 0x55b128 thru 0x55bebb (0xd94 bytes) 000296[0] 1787364596.217739 trunc from 0x897ccd to 0x29261b (0x6056b1 bytes) 000298[0] 1787364596.338724 mapread 0x1021a3 thru 0x104112 (0x1f70 bytes) 000301[1] 1787364596.390519 write 0x802700 thru 0x8034f8 (0xdf9 bytes) 000303[0] 1787364596.435173 mapwrite 0x88911f thru 0x88ab41 (0x1a23 bytes) 000307[0] 1787364596.643946 read 0x190f4a thru 0x19176a (0x821 bytes) 000309[1] 1787364596.682805 mapread 0x7dc6e0 thru 0x7dca05 (0x326 bytes) 000319[0] 1787364596.837885 mapread 0x1bed4c thru 0x1bfa66 (0xd1b bytes) 000321[0] 1787364596.875707 trunc from 0x88ab42 to 0x525079 (0x365ac8 bytes) 000323[0] 1787364596.919430 mapwrite 0x166426 thru 0x166b5c (0x737 bytes) 000324[0] 1787364597.015526 mapwrite 0x270c62 thru 0x272a66 (0x1e05 bytes) 000325[0] 1787364597.082455 mapread 0x0bab04 thru 0x0bb2bd (0x7ba bytes) 000326[0] 1787364597.092856 write 0x20250e thru 0x203cf7 (0x17ea bytes) 000328[0] 1787364597.122574 mapread 0x37fdde thru 0x38155f (0x1782 bytes) 000329[1] 1787364597.161960 mapread 0x0c7ccd thru 0x0c802e (0x362 bytes) 000330[0] 1787364597.229530 read 0x45bea6 thru 0x45d7bb (0x1916 bytes) 000332[0] 1787364597.258631 write 0x3870b9 thru 0x387289 (0x1d1 bytes) 000334[0] 1787364597.324599 trunc from 0x52507a to 0x8ca90f (0x3a5896 bytes) 000335[1] 1787364597.386560 write 0x7e87de thru 0x7e90c1 (0x8e4 bytes) 000337[0] 1787364597.445576 mapread 0x08d469 thru 0x08f130 (0x1cc8 bytes) 000340[1] 1787364597.525790 write 0x199865 thru 0x199e3b (0x5d7 bytes) 000345[1] 1787364597.640723 trunc from 0x8ca910 to 0x08c159 (0x83e7b6 bytes) 000347[0] 1787364597.707651 read 0x033d22 thru 0x034117 (0x3f6 bytes) 000348[0] 1787364597.751713 write 0x5b5c9a thru 0x5b5d0c (0x73 bytes) 000350[1] 1787364597.819006 mapwrite 0x488002 thru 0x489c9c (0x1c9b bytes) 000353[0] 1787364597.921661 trunc from 0x5b5d0d to 0x146ea8 (0x46ee64 bytes) 000356[1] 1787364598.035521 read 0x070267 thru 0x07172c (0x14c6 bytes) 000357[1] 1787364598.088516 mapread 0x133498 thru 0x1344fb (0x1064 bytes) 000358[0] 1787364598.098887 trunc from 0x146ea9 to 0x355a7b (0x20ebd3 bytes) 000359[0] 1787364598.155120 mapwrite 0x8ebc3b thru 0x8ebdfa (0x1c0 bytes) 000362[0] 1787364598.297729 write 0x1fedc3 thru 0x1ff49a (0x6d8 bytes) 000363[1] 1787364598.339705 mapwrite 0x23d41f thru 0x23e6a2 (0x1284 bytes) 000364[0] 1787364598.503154 read 0x6729c1 thru 0x6733cd (0xa0d bytes) 000365[0] 1787364598.525996 mapwrite 0x45aa2e thru 0x45c720 (0x1cf3 bytes) 000367[0] 1787364598.624627 mapread 0x4a03bd thru 0x4a0d5b (0x99f bytes) 000368[0] 1787364598.667413 read 0x3f3880 thru 0x3f38bf (0x40 bytes) 000370[1] 1787364598.718961 mapwrite 0x8134a9 thru 0x814c02 (0x175a bytes) 000372[0] 1787364598.834550 mapwrite 0x047f79 thru 0x049680 (0x1708 bytes) 000373[1] 1787364598.946042 write 0x22edc7 thru 0x23006e (0x12a8 bytes) 000374[1] 1787364598.979878 mapwrite 0x76a4e5 thru 0x76a722 (0x23e bytes) 000376[1] 1787364599.187232 trunc from 0x8ebdfb to 0x9a1371 (0xb5577 bytes) 000377[1] 1787364599.231396 trunc from 0x9a1372 to 0x2f119c (0x6b01d5 bytes) 000378[0] 1787364599.293796 mapread 0x0d7022 thru 0x0d70bf (0x9e bytes) 000380[1] 1787364599.384140 mapwrite 0x55b32a thru 0x55bb3b (0x812 bytes) 000381[0] 1787364599.471818 write 0x677ff8 thru 0x6797ee (0x17f7 bytes) 000383[1] 1787364599.543374 write 0x7a81c1 thru 0x7a9efa (0x1d3a bytes) 000387[0] 1787364599.600376 trunc from 0x7a9efb to 0x1a78b7 (0x602643 bytes) 000388[1] 1787364599.664804 write 0x7df4ca thru 0x7e09d2 (0x1509 bytes) 000389[0] 1787364599.709995 mapread 0x037f13 thru 0x039b13 (0x1c01 bytes) 000390[1] 1787364599.764397 trunc from 0x7e09d3 to 0x6dfa5c (0x100f76 bytes) 000395[0] 1787364599.924576 write 0x4fb44b thru 0x4fb978 (0x52e bytes) 000401[1] 1787364600.128521 read 0x265ba2 thru 0x2662a0 (0x6ff bytes) 000402[1] 1787364600.179375 read 0x18c418 thru 0x18c9e6 (0x5cf bytes) 000403[1] 1787364600.238393 mapread 0x20caa1 thru 0x20d784 (0xce4 bytes) 000404[0] 1787364600.299117 trunc from 0x6dfa5d to 0x1b5c7a (0x529de2 bytes) 000405[1] 1787364600.378486 trunc from 0x1b5c7b to 0x09f97c (0x1162fe bytes) 000407[0] 1787364600.475953 mapwrite 0x57dad0 thru 0x57e4b9 (0x9ea bytes) 000409[0] 1787364600.710272 mapwrite 0x17964b thru 0x179a28 (0x3de bytes) 000411[0] 1787364600.752707 mapread 0x1ed64d thru 0x1ee403 (0xdb7 bytes) 000412[0] 1787364600.791235 mapwrite 0x76b9c0 thru 0x76ba9e (0xdf bytes) 000413[1] 1787364600.889796 mapread 0x2e28c4 thru 0x2e2fd5 (0x712 bytes) 000414[1] 1787364600.945664 read 0x14a127 thru 0x14a7d0 (0x6aa bytes) 000415[0] 1787364600.965284 mapread 0x40787e thru 0x4084c4 (0xc47 bytes) 000418[1] 1787364601.008099 mapread 0x5d2f4c thru 0x5d3439 (0x4ee bytes) 000420[0] 1787364601.037634 trunc from 0x76ba9f to 0x5b23a5 (0x1b96f9 bytes) 000422[0] 1787364601.082103 mapread 0x1de74f thru 0x1dff91 (0x1843 bytes) 000430[1] 1787364601.239826 mapwrite 0x8f08e8 thru 0x8f1b8f (0x12a8 bytes) 000431[1] 1787364601.470406 write 0x91c672 thru 0x91cc48 (0x5d7 bytes) 000432[0] 1787364601.530792 trunc from 0x91cc49 to 0x9a1cef (0x850a7 bytes) 000433[0] 1787364601.618007 mapread 0x6c7f3b thru 0x6c7fb3 (0x79 bytes) 000437[1] 1787364601.739537 write 0x363348 thru 0x364f1f (0x1bd8 bytes) 000441[0] 1787364601.831932 write 0x70b17c thru 0x70bdfd (0xc82 bytes) 000442[1] 1787364601.860991 mapwrite 0x2dcdae thru 0x2de764 (0x19b7 bytes) 000443[1] 1787364601.976392 mapwrite 0x174dc4 thru 0x17694a (0x1b87 bytes) 000444[0] 1787364602.131732 read 0x6b0c07 thru 0x6b1828 (0xc22 bytes) 000449[1] 1787364602.278497 write 0x247eb3 thru 0x249cba (0x1e08 bytes) 000452[0] 1787364602.340678 write 0x98c985 thru 0x98df52 (0x15ce bytes) 000454[1] 1787364602.371877 mapwrite 0x26863d thru 0x26a2ff (0x1cc3 bytes) 000455[1] 1787364602.485686 read 0x0440ea thru 0x045110 (0x1027 bytes) 000457[1] 1787364602.526732 read 0x3d0787 thru 0x3d0b19 (0x393 bytes) 000459[1] 1787364602.570949 mapread 0x394ff5 thru 0x396b5e (0x1b6a bytes) truncating to largest ever: 0x9f0376 000463[0] 1787364602.737624 trunc from 0x9a1cf0 to 0x9f0375 (0x4e686 bytes) 000465[1] 1787364602.809463 mapread 0x6388ea thru 0x63a5b8 (0x1ccf bytes) 000466[0] 1787364602.868609 write 0x7507a8 thru 0x751036 (0x88f bytes) 000469[1] 1787364602.991551 write 0x9cba95 thru 0x9cd95f (0x1ecb bytes) 000470[1] 1787364603.033997 trunc from 0x9f0376 to 0x668750 (0x387c25 bytes) 000471[0] 1787364603.114970 mapread 0x4c0e69 thru 0x4c1d34 (0xecc bytes) 000472[1] 1787364603.162417 write 0x9ac4b4 thru 0x9ac5ee (0x13b bytes) 000474[0] 1787364603.184784 read 0x10a97c thru 0x10bb36 (0x11bb bytes) 000476[0] 1787364603.213542 read 0x764e8c thru 0x76698d (0x1b02 bytes) 000480[0] 1787364603.227671 write 0x0484f9 thru 0x04857f (0x87 bytes) 000482[1] 1787364603.263767 write 0x605d57 thru 0x607336 (0x15e0 bytes) 000483[0] 1787364603.304687 mapwrite 0x171cee thru 0x171fcf (0x2e2 bytes) 000484[1] 1787364603.415786 trunc from 0x9ac5ef to 0x080788 (0x92be66 bytes) 000486[1] 1787364603.519061 write 0x05b758 thru 0x05c62c (0xed5 bytes) 000487[1] 1787364603.563509 write 0x7ccf2f thru 0x7cd570 (0x642 bytes) 000491[0] 1787364603.630902 write 0x35c2ab thru 0x35d292 (0xfe8 bytes) 000492[0] 1787364603.674237 read 0x51d7bf thru 0x51e139 (0x97b bytes) 000497[1] 1787364603.720630 write 0x893cff thru 0x895782 (0x1a84 bytes) 000499[0] 1787364603.755658 trunc from 0x895783 to 0x3b81f1 (0x4dd591 bytes) 000500[0] 1787364603.828593 write 0x4655d9 thru 0x465666 (0x8e bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 22:10:09 (1787364609) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 22:10:13 (1787364613) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.562948 s, 14.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.25619 s, 32.7 MB/s ldlm.namespaces.MGC192.168.202.125@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a0346458000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9a034cb68000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9a034cb68000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9a034cb68000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9708 s, 8.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.96285 s, 26.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.04714 s, 34.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.84103 s, 18.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.29284 s, 11.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19891 s, 20.2 MB/s PASS 16d (53s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 22:11:06 (1787364666) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333176 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683072 s, 15.4 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 22:11:15 (1787364675) TIMEOUT PASS 16f (49s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 22:12:05 (1787364725) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 22:12:34 (1787364754) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 22:12:43 (1787364763) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 22:12:51 (1787364771) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36108 s, 11.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.65204 s, 15.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.69996 s, 9.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.89922 s, 13.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.59114 s, 10.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64267 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.30208 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.3468 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46788 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.29314 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.47984 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54113 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.7895 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.23328 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.51573 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.51159 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.96053 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18454 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.32862 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 22:13:33 (1787364813) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Aug 21 22:14:27 EDT 2026 + fsx 2 done: rc=0 Fri Aug 21 22:14:28 EDT 2026 + fsx 3 done: rc=0 Fri Aug 21 22:14:28 EDT 2026 + fsx 4 done: rc=0 Fri Aug 21 22:14:28 EDT 2026 + fsx 5 done: rc=0 Fri Aug 21 22:14:28 EDT 2026 PASS 16k (66s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 22:14:39 (1787364879) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 22:14:52 (1787364892) mmap test1: basic mmap operation (PASS, 0.17497s) mmap test2: MAP_PRIVATE not write back (PASS, 0.080059s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.0484s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5835s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.17659s) mmap test6: check mmap write/read content on two nodes (PASS, 0.16836s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2922s) mmap test11: multi-thread shared mmap (PASS, 1.658s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1707s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:15:21 (1787364921) oleg225-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (7s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:15:27 (1787364927) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:15:34 (1787364934) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 22:15:41 (1787364941) now is 1787364945 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 new atime is 1787365006 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:16:52 (1787365012) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:17:00 (1787365020) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:17:08 (1787365028) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:17:16 (1787365036) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 22:17:19 (1787365039) utime: good mknod times 1787365039 <= 1787365040 <= 1787365040 for /mnt/lustre/f26a utime: good mknod times 1787365039 <= 1787365040 <= 1787365040 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 22:17:27 (1787365047) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:17:36 (1787365056) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-21 22:17:38.000000000 -0400 Birth: 2026-08-21 22:17:37.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-21 22:17:38.000000000 -0400 Birth: 2026-08-21 22:17:37.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:17:48 (1787365068) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:17:58 (1787365078) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:18:06 (1787365086) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg225-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.125@tcp:/lustre 7542784 38912 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 22:18:30 (1787365110) -rw-r--r-- 1 root root 0 Aug 21 22:18 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:18:40 (1787365120) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43157 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:18:47 (1787365127) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 22:18:56 (1787365136) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 22:19:00 (1787365140) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69723 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47726 s, 35.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.36472 s, 22.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.8306 s, 18.5 MB/s contention_seconds 4, contention_events 25 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.92057 s, 27.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.2479 s, 23.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08298 s, 25.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.06896 s, 25.3 MB/s PASS 32c (34s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:19:35 (1787365175) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:19:39 (1787365179) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:19:43 (1787365183) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:19:46 (1787365186) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 22:19:50 (1787365190) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid in FULL state after 0 sec oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid in IDLE state after 0 sec oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid in IDLE state after 0 sec oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid in IDLE state after 0 sec oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a0346458000.ost_server_uuid in IDLE state after 0 sec oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a0346458000.ost_server_uuid in IDLE state after 0 sec PASS 34 (143s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:22:13 (1787365333) fail_loc=0x80000317 Race attempt 0 Wait for 50466 50508 for 60 sec... - open/close 748 (time 1787365350.67 total 10.01 last 74.69) - open/close 1519 (time 1787365360.67 total 20.02 last 77.09) - open/close 2284 (time 1787365370.68 total 30.02 last 76.45) - open/close 3128 (time 1787365380.69 total 40.03 last 84.36) - open/close 3966 (time 1787365390.69 total 50.03 last 83.75) total: 4000 open/close in 50.62 seconds: 79.02 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.125@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.125@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:23:26 (1787365406) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg225-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59491 s, 32.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.55042 s, 33.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7491584) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30832 s, 40.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34916 s, 38.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65913 s, 31.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65438 s, 31.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7493632) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36928 s, 38.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49352 s, 35.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64548 s, 31.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77728 s, 29.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7442432) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22729 s, 42.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7444480) after(7495680) PASS 36 (290s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:28:16 (1787365696) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 - create 1565 (time 1787365710.27 total 10.00 last 156.45) - create 3175 (time 1787365720.28 total 20.01 last 160.96) - create 4691 (time 1787365730.28 total 30.01 last 151.60) - create 6184 (time 1787365740.28 total 40.01 last 149.29) - create 7588 (time 1787365750.28 total 50.01 last 140.34) - create 8942 (time 1787365760.28 total 60.01 last 135.39) total: 10000 create in 66.84 seconds: 149.60 ops/second PASS 37 (77s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:29:34 (1787365774) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:29:41 (1787365781) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:29:49 (1787365789) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 22:29:57 (1787365797) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:30:06 (1787365806) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:30:24 (1787365824) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:30:42 (1787365842) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:31:00 (1787365860) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:31:17 (1787365877) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:31:29 (1787365889) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:31:42 (1787365902) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:31:55 (1787365915) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:32:07 (1787365927) Conflict PASS 41e (18s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:32:25 (1787365945) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:32:39 (1787365959) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:32:52 (1787365972) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 22:33:06 (1787365986) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1069s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:50:55 (1787367055) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:51:10 (1787367070) open: File exists Conflict PASS 42b (18s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:51:28 (1787367088) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:51:44 (1787367104) Conflict PASS 42d (17s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:52:02 (1787367122) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:52:18 (1787367138) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:52:33 (1787367153) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:52:48 (1787367168) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:53:04 (1787367184) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:54:24 (1787367264) Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:54:39 (1787367279) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:54:55 (1787367295) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:55:09 (1787367309) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:55:25 (1787367325) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (16s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:55:41 (1787367341) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (18s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:55:59 (1787367359) Conflict PASS 43h (16s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:56:16 (1787367376) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:56:20 (1787367380) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (137s) == sanityn test 43k: unlink vs create ==================== 22:58:38 (1787367518) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1297s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 23:20:16 (1787368816) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Aug 21 23:20:25 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 23:20:30 (1787368830) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 23:20:48 (1787368848) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (17s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 23:21:05 (1787368865) Conflict PASS 44d (16s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 23:21:21 (1787368881) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 23:21:36 (1787368896) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 23:21:49 (1787368909) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 23:22:03 (1787368923) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 23:22:18 (1787368938) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 23:22:22 (1787368942) PASS 45a (130s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 23:24:32 (1787369072) Conflict PASS 45b (16s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 23:24:48 (1787369088) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 23:25:03 (1787369103) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 23:25:18 (1787369118) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 23:25:33 (1787369133) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 23:25:48 (1787369148) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 23:26:03 (1787369163) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 23:26:15 (1787369175) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 23:26:19 (1787369179) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1265s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:47:24 (1787370444) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:47:39 (1787370459) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:47:53 (1787370473) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (11s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:48:05 (1787370485) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:48:19 (1787370499) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:48:32 (1787370512) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:48:47 (1787370527) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:49:01 (1787370541) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:49:13 (1787370553) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:49:16 (1787370556) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:49:20 (1787370560) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:49:23 (1787370563) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:49:26 (1787370566) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:49:29 (1787370569) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:49:31 (1787370571) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:49:35 (1787370575) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:49:38 (1787370578) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0753444 s, 136 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:49:49 (1787370589) 0+1 records in 0+1 records out 158 bytes copied, 0.0236201 s, 6.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218401) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:49:59 (1787370599) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0125559 s, 81.6 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802641 s, 13.1 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:50:21 (1787370621) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0170033 s, 60.2 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:50:34 (1787370634) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326279 s, 32.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:50:42 (1787370642) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 23:50:51 (1787370651) llite.lustre-ffff9a0346458000.xattr_cache=1 llite.lustre-ffff9a034cb68000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:50:53.000000000 -0400 Modify: 2026-08-21 23:50:53.000000000 -0400 Change: 2026-08-21 23:50:53.000000000 -0400 Birth: 2026-08-21 23:50:52.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:51:00.000000000 -0400 Modify: 2026-08-21 23:51:01.000000000 -0400 Change: 2026-08-21 23:51:01.000000000 -0400 Birth: 2026-08-21 23:50:59.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:51:06.000000000 -0400 Modify: 2026-08-21 23:51:06.000000000 -0400 Change: 2026-08-21 23:51:06.000000000 -0400 Birth: 2026-08-21 23:51:05.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:51:06.000000000 -0400 Modify: 2026-08-21 23:51:07.000000000 -0400 Change: 2026-08-21 23:51:07.000000000 -0400 Birth: 2026-08-21 23:51:05.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:51:12.000000000 -0400 Modify: 2026-08-21 23:51:13.000000000 -0400 Change: 2026-08-21 23:51:13.000000000 -0400 Birth: 2026-08-21 23:51:11.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 23:51:12.000000000 -0400 Modify: 2026-08-21 23:51:13.000000000 -0400 Change: 2026-08-21 23:51:13.000000000 -0400 Birth: 2026-08-21 23:51:11.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 23:51:23 (1787370683) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 23:51:35 (1787370695) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (12s) == sanityn test 55c: rename vs unlink orphan target dir == 23:51:47 (1787370707) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (22s) == sanityn test 55d: rename file vs link ================= 23:52:09 (1787370729) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:52:25 (1787370745) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:52:28 (1787370748) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 23:52:48 (1787370768) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (26s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:53:14 (1787370794) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1787370796 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.586914 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (114s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:55:08 (1787370908) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1787370916 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.992473 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (138s) == sanityn test 60: Verify data_version behaviour ======== 23:57:26 (1787371046) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 23:57:36 (1787371056) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:57:45 (1787371065) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:57:56 (1787371076) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0135619 s, 4.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:58:01 (1787371081) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.015827 s, 4.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:58:05 (1787371085) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:58:09 (1787371089) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0129128 s, 5.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:58:14 (1787371094) llite.lustre-ffff9a0346458000.xattr_cache=1 llite.lustre-ffff9a034cb68000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:58:23 (1787371103) llite.lustre-ffff9a0346458000.xattr_cache=1 llite.lustre-ffff9a034cb68000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:58:31 (1787371111) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0210997 s, 48.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0182647 s, 56.1 kB/s 231210: get lock2 231214: get lock1 231214: done 231214: get lock2 231210: done 231210: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231210: cannot get lock1: Resource deadlock avoided 231214: done 231214: exit rc=0 231210: done 231210: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:58:43 (1787371123) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0695129 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381823 s, 27.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231796 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 23:58:55 (1787371135) opening files: ........ closing files: ........ PASS 76 (81s) == sanityn test 77a: check FIFO NRS policy =============== 00:00:16 (1787371216) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.564417 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104863 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884939 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155372 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121996 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141825 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248236 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233041 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103085 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273569 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321622 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349662 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361943 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187766 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114892 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281637 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137871 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10283 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342111 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423877 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106288 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670565 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758317 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138348 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823412 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811624 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729852 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157036 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074431 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906829 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839538 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127712 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722474 s, 145 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (12s) == sanityn test 77b: check CRR-N NRS policy ============== 00:00:28 (1787371228) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.741617 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108462 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129299 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926015 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20467 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132916 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162222 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202806 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174592 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126925 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184408 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182773 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155545 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264006 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263142 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161753 s, 6.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201243 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101566 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595524 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144101 s, 72.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491912 s, 21.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149853 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520468 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136456 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173196 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368199 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707512 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244783 s, 42.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114084 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226723 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114204 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158483 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191407 s, 54.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.69311 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108999 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119592 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27009 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192623 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224505 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332057 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336572 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25872 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352108 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338304 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314572 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357517 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334739 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31224 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367676 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396466 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403484 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519492 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547264 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016037 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785835 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486055 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110579 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00315403 s, 332 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266598 s, 393 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119451 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245853 s, 427 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00269715 s, 389 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989572 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143054 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13822 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346021 s, 30.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (15s) == sanityn test 77c: check ORR NRS policy ================ 00:00:43 (1787371243) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.38566 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107501 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164045 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212201 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193983 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134365 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197555 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292996 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401084 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41828 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380955 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419509 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407518 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39614 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496725 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41115 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.547241 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102705 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605279 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626968 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681181 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750469 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692383 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728324 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075193 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761818 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785527 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791339 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864141 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042323 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860483 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087017 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493983 s, 21.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.612954 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830274 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875179 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140435 s, 7.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263106 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261855 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297174 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253284 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278919 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315341 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237529 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3048 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331302 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332638 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299439 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296234 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380813 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10069 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513307 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109746 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533338 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125235 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955225 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572117 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604692 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126253 s, 8.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106144 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632856 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123586 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100741 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745157 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944622 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101919 s, 10.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518673 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172833 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107718 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297478 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303651 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183943 s, 5.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231657 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317961 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252457 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276131 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331459 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282876 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363785 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418928 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325086 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402952 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490601 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358924 s, 292 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101087 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917821 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061074 s, 172 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149166 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542063 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491514 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058746 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00280994 s, 373 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843097 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158343 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153808 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628057 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228461 s, 45.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101613 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228604 s, 45.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 00:01:00 (1787371260) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317156 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633203 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698197 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120772 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139623 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238343 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296074 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282233 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236553 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399307 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30003 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29482 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334311 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117487 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357434 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451562 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691224 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397878 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255066 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437784 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199061 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163308 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248642 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027767 s, 37.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618588 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455918 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04031 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905309 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049929 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470873 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666489 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619664 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173304 s, 60.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.50206 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155128 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836058 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154183 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205207 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130753 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128251 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113921 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235851 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35841 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255925 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160084 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220644 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217595 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392868 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171455 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457343 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714282 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603952 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752399 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053495 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542248 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604287 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533401 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589347 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516556 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703135 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704594 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973246 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716482 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059312 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669568 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662976 s, 15.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571648 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693341 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814802 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804008 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135345 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218986 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223037 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183303 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182325 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187634 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205984 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310711 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318986 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290035 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337765 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228688 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25856 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403015 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497175 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670383 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820275 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077722 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501312 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584288 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733573 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946518 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823479 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972598 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759318 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890191 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907346 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832197 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986542 s, 10.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 00:01:18 (1787371278) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.25@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.637496 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131975 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101009 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165616 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197907 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170571 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289944 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366427 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284557 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341079 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355252 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416374 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423468 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447157 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415231 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507558 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509041 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013958 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785456 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290418 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101607 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910368 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683032 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959564 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119703 s, 87.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681225 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665337 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00686769 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175216 s, 59.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075934 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195969 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110589 s, 94.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135029 s, 77.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.371319 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368629 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963441 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103203 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100179 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154676 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176524 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145941 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21306 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150609 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190389 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159864 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284022 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2502 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242146 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323567 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371127 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238449 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820784 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332676 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314854 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253958 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419165 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041662 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556965 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916885 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617212 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472922 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456148 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485431 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484413 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692595 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505822 s, 20.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.656106 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120687 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701115 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196387 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115507 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751528 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11639 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571161 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155074 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113759 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329918 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293858 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370279 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337369 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379194 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422803 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302102 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546312 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00893067 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270519 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355874 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099204 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842888 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108452 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940011 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1025 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145918 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115238 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210329 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161702 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157318 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192146 s, 5.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168939 s, 6.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.497476 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117819 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132999 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153631 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160944 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701116 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957399 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936211 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538993 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166955 s, 6.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226026 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302004 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431662 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269458 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30869 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389032 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478399 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883101 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688025 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271457 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355492 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215189 s, 48.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225247 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014453 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292565 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293605 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240588 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039198 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301785 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1515 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1244 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246409 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178812 s, 5.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (27s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:01:45 (1787371305) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.926398 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141682 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156003 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12207 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101923 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40829 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245765 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470921 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136874 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329939 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.556228 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422481 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527544 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206013 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277175 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.568208 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257208 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740459 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471943 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528857 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962401 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105002 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503197 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765728 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11982 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700544 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730798 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112717 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965988 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110499 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977276 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149803 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112087 s, 9.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.526527 s, 31.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108267 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175439 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169003 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29242 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259607 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141176 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237899 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385856 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373644 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229704 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445619 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207928 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428077 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320352 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215517 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288359 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613596 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701325 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105678 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601043 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093611 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108393 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629155 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721035 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07201 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110537 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125443 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116616 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089635 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913414 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137826 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121452 s, 8.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571078 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082189 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169862 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216682 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280607 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161607 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158294 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191806 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199114 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168681 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426277 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374627 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35554 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412802 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362769 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233845 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126429 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864946 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00499831 s, 210 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102238 s, 103 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120053 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127332 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153967 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991296 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635939 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595128 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508645 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776664 s, 135 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570791 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056328 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093769 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046887 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470134 s, 22.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.500884 s, 33.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868933 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797657 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138466 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106177 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484891 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119448 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679434 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203027 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281945 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273256 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219819 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335522 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415456 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39775 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497015 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.56961 s, 1.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028822 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258243 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612332 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266505 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714007 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072418 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308596 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722551 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906144 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507997 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600207 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574653 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988851 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983527 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596298 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718862 s, 14.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (32s) == sanityn test 77g: Change TBF type directly ============ 00:02:17 (1787371337) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:02:29 (1787371349) oleg225-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg225-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 oleg225-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg225-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22 oleg225-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg225-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22 oleg225-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg225-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 00:02:42 (1787371362) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:03:04 (1787371384) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663736 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055377 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145424 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153755 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109902 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136665 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228341 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192579 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199069 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258071 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249024 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249775 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23446 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245319 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273922 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254926 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279136 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345253 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783055 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612366 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480345 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528661 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104127 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600128 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612564 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770424 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526095 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888831 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064436 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889179 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934687 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724556 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712558 s, 14.7 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32829 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6234 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:04:09 (1787371449) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32328 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144223 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180989 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157459 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104473 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135765 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277447 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261303 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2721 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312847 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342066 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331844 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367471 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38974 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406134 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26146 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445961 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780959 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747883 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586084 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886384 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734445 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775408 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968407 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637728 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785179 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873744 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771985 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746206 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788044 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892113 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819837 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926666 s, 11.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9892 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9535 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35118 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109052 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099648 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196681 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384122 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175156 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217968 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137083 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248587 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202452 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231068 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197827 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302829 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24292 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115192 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216719 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217632 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760594 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086899 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754581 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108871 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112997 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783967 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104574 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133577 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901856 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117034 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089173 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121365 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110016 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989086 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160408 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121181 s, 8.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9657 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9587 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (157s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:06:45 (1787371605) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31423 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753585 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051899 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109329 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13486 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138307 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148555 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852204 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15178 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230548 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233911 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233043 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217006 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234138 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287512 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245626 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263018 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741928 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499705 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659707 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598179 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101491 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696526 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622562 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850531 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112725 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110401 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813458 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799762 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777634 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221753 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089567 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107406 s, 9.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0098 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8031 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.36842 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954978 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183934 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239488 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300996 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127559 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236447 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232156 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287212 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256296 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239986 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338225 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301839 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345401 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327117 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354381 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359549 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488654 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693085 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451585 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309072 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695806 s, 151 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105821 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14544 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083798 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136219 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930314 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425138 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324129 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996066 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997111 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159632 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165402 s, 6.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2923 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9694 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (155s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:09:20 (1787371760) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.611943 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668704 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588292 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611026 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992777 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684541 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115796 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122707 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996306 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188321 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205125 s, 5.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205951 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130458 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224423 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153539 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191274 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2839 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658674 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114799 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106247 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991627 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703601 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10226 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13311 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893285 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146084 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096487 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19974 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880589 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128332 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10776 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116859 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185715 s, 5.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.88211 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97893 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.25@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.25@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.536532 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056802 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696951 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897218 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099405 s, 10.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101638 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05652 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887093 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161041 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116117 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110041 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130911 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151965 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21465 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119156 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226543 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157118 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210257 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366663 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263662 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357274 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063887 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016857 s, 62.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383527 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05597 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059714 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639028 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491872 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549163 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476889 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893989 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506505 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493945 s, 21.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.00723 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87662 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.25@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.612377 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941876 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962961 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863643 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138007 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148205 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126902 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134199 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129464 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634822 s, 16.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382232 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365482 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397139 s, 2.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441737 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476424 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487705 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49587 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333049 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372184 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412875 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711708 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383499 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046894 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418329 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886542 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637078 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646898 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547404 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896196 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689443 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686402 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074611 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964629 s, 10.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.04985 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20675 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.645785 s, 26.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704724 s, 14.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778604 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121097 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125963 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141378 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173962 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301012 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702782 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105402 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949209 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254558 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983227 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289176 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308605 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212241 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24195 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104676 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705461 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363477 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113935 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072159 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501591 s, 20.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349698 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480589 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545745 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493444 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690432 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465066 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641605 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114685 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164388 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130007 s, 80.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5131 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91339 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35326 s, 14.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9804 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05881 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121018 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121556 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153713 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178345 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200119 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26389 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381964 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287572 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383468 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312302 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426822 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306714 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463205 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32508 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323963 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516058 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041116 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430997 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494139 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587283 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541152 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112538 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461298 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066368 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746424 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127554 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079644 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727242 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751056 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822836 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100363 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11423 s, 9.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3331 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6571 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.513916 s, 32.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875124 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100794 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106082 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853867 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203939 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245326 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328333 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21776 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313582 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25089 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251409 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301016 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243879 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292397 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290462 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315119 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239902 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393029 s, 26.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318074 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194476 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751363 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338412 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440992 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335951 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587429 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472742 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04787 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537553 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541105 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581055 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059877 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635431 s, 16.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.53466 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99733 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.79748 s, 21.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588477 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969194 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16302 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125479 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148283 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190847 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262122 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263293 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251463 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302175 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226475 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310106 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266066 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278722 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340447 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357594 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679344 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879937 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471491 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660846 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581668 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627484 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513522 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764382 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780853 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100742 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955118 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964729 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936447 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102191 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776383 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120505 s, 8.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4022 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98946 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5182 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1526 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (449s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:16:49 (1787372209) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.748329 s, 22.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694189 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121015 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119804 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264916 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208672 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256224 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329108 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307679 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305544 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325611 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311656 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386078 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348303 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39492 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329701 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446082 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258268 s, 406 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261177 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517686 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881325 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064645 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879062 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558913 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320464 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106009 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064384 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573236 s, 18.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118273 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822213 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580221 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119142 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909128 s, 11.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.1126 s, 14.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99606 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:17:41 (1787372261) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.25@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.25@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.532105 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773562 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826199 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137421 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187742 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196702 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205952 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209752 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229399 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212016 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231403 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279759 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233072 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298104 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262576 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27419 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304465 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794759 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429497 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423161 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556483 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805542 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421655 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470401 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995567 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567547 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625822 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746033 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664538 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818766 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660224 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820416 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991328 s, 10.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.97907 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96253 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (54s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:18:36 (1787372316) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.25@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.838878 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455675 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767712 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107234 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118986 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16637 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220491 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25049 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203505 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306603 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278901 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346785 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359513 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351454 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35086 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374181 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43063 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13876 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150557 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164077 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190076 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159205 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18846 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189368 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212009 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228824 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225604 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243857 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12066 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109771 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223948 s, 4.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9a0346458000.lock_unused_count=9 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253508 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271324 s, 3.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg225-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.74715 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41824 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (46s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:19:21 (1787372361) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.74135 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591194 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098674 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656387 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390937 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105547 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807265 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145721 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16942 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260755 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132075 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281068 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09273 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20827 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124527 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30505 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200539 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013215 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804268 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148053 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00283066 s, 370 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114553 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153234 s, 68.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00850226 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00864568 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843287 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888012 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428673 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911879 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736762 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305025 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918225 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886709 s, 11.8 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9739 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92166 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.5813 s, 13.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93092 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:20:58 (1787372458) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15124 s, 14.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726022 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100198 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180725 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198157 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252044 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239209 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20183 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202679 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183407 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220142 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242583 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256182 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316208 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357737 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340744 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319956 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074231 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11655 s, 9.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794093 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818887 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890815 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976339 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763372 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088989 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944916 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749789 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101272 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977722 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119734 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110741 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934822 s, 11.2 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8431 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9532 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (78s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:22:16 (1787372536) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.536985 s, 31.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657877 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684486 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186994 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259707 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291579 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263704 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304495 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290728 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281662 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28499 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347973 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322056 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337761 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31213 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367576 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333898 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9a0346458000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476051 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586074 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735733 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743834 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320889 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805455 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798395 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780617 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880707 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091721 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851495 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106403 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139307 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093791 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128777 s, 8.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123824 s, 8.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.29791 s, 12.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91532 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg225-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.737879 s, 22.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875376 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718527 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515295 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170897 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200452 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128016 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269601 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32475 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247881 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22004 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339148 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211742 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414328 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25949 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312865 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438914 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100571 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403501 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379464 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768784 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461037 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612011 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495413 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566367 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801911 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797485 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669573 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740264 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070522 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759286 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790086 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850014 s, 12.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg225-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg225-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4314 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91516 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0538 s, 9.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92355 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn